TLDR: Hands-on guide to building and operating a production-grade 5-container Slurm cluster locally with Docker Compose (~1GB RAM), featuring Munge auth, SlurmDBD accounting, GPU simulation (GRES), and CLI mastery.

Introduction: From Theory to a Living HPC Cluster

In Slurm 101, we explored the architectural theory of Slurm and why it powers the world's largest AI supercomputers.

However, reading theory alone leaves critical operational questions unanswered:

How does Munge handle cryptographic authentication handshakes between distributed daemons?
How does SlurmDBD track and persist GPU-hours into a relational database?
How does Slurm automatically isolate resources and inject $CUDA_VISIBLE_DEVICES into job contexts?

In this guide, we will build a fully functional multi-node Slurm cluster on your local machine (1 PC or Laptop) using Docker Compose. Despite consuming only ~1GB of RAM, this local lab accurately mirrors 95% of real-world production HPC architectures.

Companion Repository: All configuration files, automated Make targets, and comprehensive documentation are open-sourced at github.com/BlackMetalz/slurm-101.

Phase 1: Cluster Architecture & The 4 Vital Pillars

Our local cluster consists of 5 containers connected via an isolated bridge network (slurm_net):

[ Your Local Host Terminal (Native Login Node) ]
                   │
                   │ (Direct RPC: localhost:6817 / 6819)
                   ▼
+-----------------------------------------------------------------------------------+
|                            Docker Bridge Network: slurm_net                       |
|                                                                                   |
|  +--------------------+        +--------------------+      +--------------------+ |
|  |     mariadb        | <----> |      slurmdbd      | <--> |     slurmctld      | |
|  | (Accounting DB)    |        |   (Accounting)     |      | (Controller/Sched) | |
|  +--------------------+        +--------------------+      +--------------------+ |
|                                                                      |            |
|                                       +------------------------------+            |
|                                       v                              v            |
|                             +--------------------+         +--------------------+ |
|                             |      cnode01       |         |      cnode02       | |
|                             |  (Partition: cpu)  |         |  (Partition: gpu)  | |
|                             | (2 CPUs / 1GB RAM) |         |(2 CPUs + 2x A100)  | |
|                             +--------------------+         +--------------------+ |
+-----------------------------------------------------------------------------------+

The 4 Vital Pillars of HPC Infrastructure:

Pillar	Key Component	Why It Matters (The "Make or Break" Rule)
1. Cryptographic Auth	`/etc/munge/munge.key`	Munge provides ticket-based auth without passwords. Every node must share an identical key with strict `0400` permissions (owned by `munge:munge`). Open permissions (`0644`/`0777`) will crash Munge immediately.
2. Identity Consistency	User `hpcuser` (UID: `1001`)	Slurm runs jobs under the submitter's UID/GID. If UID `1001` exists on the login node but maps to someone else on worker nodes, jobs fail with `Permission Denied`.
3. Shared POSIX Storage	Volume `/shared`	Simulates shared file systems (NFS/Lustre). Batch scripts and output logs (`job_%j.out`) must reside on a shared mount path accessible by all nodes.
4. Network Topology	Docker Bridge DNS	Daemons resolve each other by hostname (`slurmctld`, `slurmdbd`, `cnode01`, `cnode02`).

Phase 2: Deploying the Mini-Cluster with Docker Compose

To optimize memory footprint and build speed, we use a single base image based on Ubuntu 22.04 LTS containing all necessary Slurm binaries.

slurm-101/
├── Dockerfile                   # Unified base image (Ubuntu 22.04 + Slurm + Munge)
├── docker-compose.yml           # 5-container cluster specification
├── entrypoint.sh                # Startup daemon lifecycle, Munge sync, and permission guard
├── Makefile                     # 1-Click shortcuts (make up, make test, make shell)
├── config/                      # Mounted configuration files
│   ├── slurm.conf               # Master cluster, partition, and node definitions
│   ├── slurmdbd.conf            # Accounting daemon configuration
│   └── gres.conf                # Generic resource configuration (Dummy A100 GPUs)
└── shared/                      # Simulated Shared POSIX Storage (/shared)

Key Configuration Highlights

Instead of dumping hundreds of lines of code, here are the critical configuration parameters:

1. In `slurm.conf`:

# Container-friendly process tracking (no systemd/dbus requirements)
ProctrackType=proctrack/linuxproc
TaskPlugin=task/none
SlurmdParameters=config_overrides

# Fine-grained consumable resource tracking (Cores + Memory + GPUs)
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
GresTypes=gpu

# Accounting connection to SlurmDBD
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=slurmdbd
AccountingStoragePort=6819

# Node & Partition definitions
NodeName=cnode01 CPUs=2 RealMemory=1000 State=UNKNOWN
NodeName=cnode02 CPUs=2 RealMemory=1000 Gres=gpu:a100:2 State=UNKNOWN

PartitionName=cpu Nodes=cnode01 Default=YES MaxTime=INFINITE State=UP
PartitionName=gpu Nodes=cnode02 Default=NO MaxTime=INFINITE State=UP
PartitionName=all Nodes=cnode01,cnode02 Default=NO MaxTime=INFINITE State=UP

2. In `docker-compose.yml`:

Ports Exposed: 6817:6817 (slurmctld) and 6819:6819 (slurmdbd) to allow native host access.
Compute Node Privileges: privileged: true on cnode01 and cnode02 so slurmd can switch UIDs and create process namespaces.

Full Source Files on GitHub:

Phase 3: Database Accounting & Multi-Tenancy (`sacctmgr`)

In real AI organizations, compute clusters are shared across multiple teams (NLP, Computer Vision, Data Platform). Slurm models multi-tenancy as an Association Hierarchy:

                             [ Cluster: homelab ]
                                      │
              ┌───────────────────────┴───────────────────────┐
              ▼                                               ▼
   [ Account: ai-team ]                              [ Account: data-team ]
   (Org: AI-Research)                                (Org: Data-Engineering)
              │                                               │
       ┌──────┴──────┐                                        ▼
       ▼             ▼                                  [ User: bob ]
[ User: kienlt ] [ User: alice ]

Initializing Accounting via `sacctmgr`:

Run the one-time initialization script (01_init_cluster.sh):

# 1. Register Cluster
sacctmgr -i add cluster homelab

# 2. Create Accounts (Teams / Cost Centers)
sacctmgr -i add account ai-team Description="AI/ML Team" Organization="AI-Org"
sacctmgr -i add account data-team Description="Data Team" Organization="Data-Org"

# 3. Create Users and Bind to Default Accounts
sacctmgr -i add user kienlt DefaultAccount=ai-team AdminLevel=Administrator
sacctmgr -i add user hpcuser DefaultAccount=ai-team

# 4. Inspect Active Associations
sacctmgr list association format=Cluster,Account,User,Partition,Share,GrpTRES,MaxJobs

Enforcing Resource Limits & QoS Quotas:

Prevent individual engineers or teams from monopolizing cluster hardware:

# Limit user alice to a maximum of 5 concurrent running jobs:
sacctmgr modify user alice set MaxJobs=5

# Restrict the entire ai-team to a maximum of 8 concurrent GPUs:
sacctmgr modify account ai-team set GrpTRES=gres/gpu=8

For advanced QoS and limit enforcement patterns, see the sacctmgr Reference Guide.

Phase 4: GPU Cluster Simulation via Generic Resources (GRES)

How can you practice submitting multi-GPU jobs, testing --gres=gpu:1, and verifying $CUDA_VISIBLE_DEVICES injection on a machine without physical NVIDIA GPUs?

Solution: Use Slurm's Generic Resources (GRES) mechanism with simulated character devices!

[ Job Request: sbatch --gres=gpu:1 ]
                 │
                 ▼
      [ slurmctld Scheduler ]
                 │ (Selects available GPU slot: Device 0)
                 ▼
       [ Worker Node: cnode02 ]
                 │
                 ├── Locks /dev/nvidia0
                 └── Injects: CUDA_VISIBLE_DEVICES=0

How It Works:

Device Creation: In entrypoint.sh, container cnode02 creates two character devices: bash mknod -m 666 /dev/nvidia0 c 195 0 2>/dev/null || true mknod -m 666 /dev/nvidia1 c 195 1 2>/dev/null || true
Resource Mapping: In config/gres.conf, these devices are declared as two NVIDIA A100 GPUs: ini NodeName=cnode02 Name=gpu Type=a100 File=/dev/nvidia[0-1]
Automatic Allocation:
Request 1 GPU: srun -p gpu --gres=gpu:1 bash -c 'echo $CUDA_VISIBLE_DEVICES' -> Output: 0
Request 2 GPUs: srun -p gpu --gres=gpu:2 bash -c 'echo $CUDA_VISIBLE_DEVICES' -> Output: 0,1

In production, engineers do not docker exec into cluster containers. You can configure your physical host machine as a native Slurm Login Node to run sinfo, squeue, and sbatch directly from your regular terminal:

[ Your Physical Host Terminal ]
  • Native Commands: sinfo, squeue, sbatch
  • Local Config: /etc/slurm/slurm.conf & /etc/munge/munge.key
                 │
                 │ Direct RPC to localhost:6817
                 ▼
[ Docker Cluster (slurmctld:6817) ]

The setup requires 3 key steps:

Install slurm-client and munge on your host OS (sudo apt install -y slurm-client munge).
Synchronize /etc/munge/munge.key (with strict 0400 permissions) and /etc/slurm/slurm.conf from the cluster.
Add DNS mapping 127.0.0.1 slurmctld slurmdbd to /etc/hosts.

For the complete step-by-step installation walkthrough, verification commands, and permissions guide, see HOST_LOGIN_NODE_GUIDE.md.

Phase 6: Slurm CLI Toolkit & Workflow Mastery

The Slurm CLI toolkit is structured into 5 operational domains:

                                 [ SLURM CLI TOOLKIT ]
                                           │
  ┌───────────────────┬────────────────────┼───────────────────┬───────────────────┐
  ▼                   ▼                    ▼                   ▼                   ▼
[ Cluster & Nodes ] [ Job Execution &    [ Active Queue &    [ Accounting &      [ Administration &
• sinfo               Interactive ]        Job Control ]       Profiling ]         Configuration ]
                    • sbatch             • squeue            • sacct             • scontrol
                    • srun               • scancel           • sstat             • sacctmgr
                    • salloc                                 • sreport

1. `sinfo` — Inspecting Nodes & Partitions

Reference: sinfo Guide

sinfo                 # Summarized view of partitions and states
sinfo -N -l            # Detailed hardware specifications per node
sinfo -R               # View reasons why nodes are down or draining

Node State Reference:

idle: 100% available and healthy.
alloc: 100% of CPUs/GPUs allocated to running jobs.
mix: Partially occupied (some CPUs free, some busy).
drain: Node taken offline for maintenance (existing jobs continue until completion).

2. `squeue` & `scancel` — Active Queue & Job Control

References: squeue Guide | scancel Guide

# Queue Inspection
squeue                # View all active and pending jobs
squeue -u $USER       # Filter jobs for your user
squeue -i 2           # Live streaming refresh every 2 seconds

# Targeted Job Cancellation
scancel 105                  # Cancel specific Job ID
scancel -u $USER             # Cancel ALL jobs owned by user
scancel -u $USER -t PENDING  # Cancel only PENDING jobs (leave running jobs untouched)
scancel --signal=SIGUSR1 105 # Send graceful checkpointing signal before stopping

Common Pending Reasons (NODELIST(REASON)):

(Resources): All requested CPUs/GPUs are busy; job will run when hardware frees up.
(Priority): Higher-priority jobs are ahead in the queue.
(ReqNodeNotAvail): Target hardware is offline or draining.
(Dependency): Waiting for a prerequisite job to finish first.

3. `sbatch`, `srun` & `salloc` — Job Execution (Batch & Interactive)

References: sbatch Guide | srun Guide | salloc Guide

A. Asynchronous Batch Execution (`sbatch`):

Create a batch script train.sh:

#!/usr/bin/env bash
#SBATCH --job-name=llm_finetune
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --gres=gpu:a100:1             # Request 1x A100 GPU
#SBATCH --time=02:00:00
#SBATCH --output=/shared/job_%j.out   # %j is automatically replaced with Job ID
#SBATCH --error=/shared/job_%j.err

echo "Job ${SLURM_JOB_ID} running on $(hostname)"
echo "Allocated GPU: ${CUDA_VISIBLE_DEVICES}"
python3 -c "import time; print('Training model...'); time.sleep(10)"

Submit with sbatch train.sh (outputs Submitted batch job 105).

B. Synchronous & Interactive Debugging (`srun`):

# Run command in parallel across all nodes simultaneously:
srun -p all -N 2 hostname

# Open an interactive PTY debugging shell directly on a GPU node:
srun -p gpu --gres=gpu:1 --pty bash -i

C. Interactive Resource Reservation (`salloc`):

Hold an interactive resource allocation open and run multiple consecutive commands without re-queuing:

salloc -p gpu --gres=gpu:1 -t 01:00:00

4. `sacct`, `sstat` & `sreport` — Accounting, Profiling & Auditing

References: sacct Guide | sstat Guide | sreport Guide

# Historical Accounting: Query past jobs, exit codes, and peak memory usage (MaxRSS):
sacct --format=JobID,JobName,Partition,State,ExitCode,Elapsed,MaxRSS

# Real-Time Profiling: Monitor live memory usage of running jobs to detect leaks before OOM:
sstat -j 105.batch --format=JobID,MaxRSS,AveCPU,AveDiskRead

# Auditing: Aggregate GPU-hours consumed per team over the last 30 days:
sreport cluster UserUtilizationByAccount Start=now-30days --tres=gres/gpu -t Hours

5. `scontrol` & `sacctmgr` — Cluster Administration & Multi-Tenancy

References: scontrol Guide | sacctmgr Guide

# Admin: Reload slurm.conf on the fly without restarting daemons
scontrol reconfigure

# Admin: Take a node offline for maintenance and resume it
scontrol update NodeName=cnode01 State=DRAIN Reason="Hardware upgrade"
scontrol update NodeName=cnode01 State=RESUME

# Database: Inspect multi-tenancy associations and QoS quotas
sacctmgr list association format=Cluster,Account,User,Partition,Share,GrpTRES,MaxJobs

Phase 7: Battle-Tested Troubleshooting (War Stories)

Building containerized Slurm clusters often triggers subtle edge cases. Below are 5 real-world debugging lessons encountered during development:

Symptom / Log Error	Root Cause & Resolution
unpack_header: protocol_version not supported (Zero Bytes)	Cause: Version mismatch between client and server. Fix: Synchronize Slurm versions (e.g., ensure both host and containers run v21.08).
error: cgroup_dbus_attach_to_scope: cannot connect to dbus system daemon: /run/dbus/system_bus_socket	Cause: Slurm v23+ `cgroup/v2` plugin requires systemd D-Bus to create process scopes in Docker. Fix: Set `ProctrackType=proctrack/linuxproc` and `TaskPlugin=task/none` in `slurm.conf`.
Node configuration differs from hardware (CPUs=2:24)	Cause: Host machine physical CPU count exceeds container config. Fix: Enable `SlurmdParameters=config_overrides` in `slurm.conf`.
Invalid GRES record: duplicate device file name (/dev/null)	Cause: Duplicate file paths mapped to multiple GPUs. Fix: Create distinct character devices (`/dev/nvidia0`, `/dev/nvidia1`) via `mknod`.
Can not recover assoc_usage state, incompatible version	Cause: Binary state files in `/var/spool/slurmctld` created by newer Slurm. Fix: Wipe state volume via `docker compose down -v`.

For deep-dive root cause analyses and remediation scripts, see TROUBLESHOOTING.md.

Conclusion

By completing this hands-on lab, you have:

Built a production-grade 5-node Slurm cluster featuring Controller orchestration, Worker nodes, Database accounting, and GPU GRES simulation.
Mastered Munge ticket-based authentication, cgroup process isolation, and POSIX shared storage tiers.
Transformed your physical machine into a native Slurm Login Node and gained proficiency across the complete CLI toolkit.

All source code, automated Make targets, and comprehensive command documentation are available at: https://github.com/BlackMetalz/slurm-101

Happy Clustering!

Slurm 202: Hands-on Multi-Node Cluster with Docker & CLI Mastery

Introduction: From Theory to a Living HPC Cluster

Phase 1: Cluster Architecture & The 4 Vital Pillars

The 4 Vital Pillars of HPC Infrastructure:

Phase 2: Deploying the Mini-Cluster with Docker Compose

Key Configuration Highlights

1. In `slurm.conf`:

2. In `docker-compose.yml`:

Phase 3: Database Accounting & Multi-Tenancy (`sacctmgr`)

Initializing Accounting via `sacctmgr`:

Enforcing Resource Limits & QoS Quotas:

Phase 4: GPU Cluster Simulation via Generic Resources (GRES)

How It Works:

Phase 6: Slurm CLI Toolkit & Workflow Mastery

1. `sinfo` — Inspecting Nodes & Partitions

2. `squeue` & `scancel` — Active Queue & Job Control

3. `sbatch`, `srun` & `salloc` — Job Execution (Batch & Interactive)

A. Asynchronous Batch Execution (`sbatch`):

B. Synchronous & Interactive Debugging (`srun`):

C. Interactive Resource Reservation (`salloc`):

4. `sacct`, `sstat` & `sreport` — Accounting, Profiling & Auditing

5. `scontrol` & `sacctmgr` — Cluster Administration & Multi-Tenancy

Phase 7: Battle-Tested Troubleshooting (War Stories)

Conclusion

Published

Category

Tags

Contact

Introduction: From Theory to a Living HPC Cluster

Phase 1: Cluster Architecture & The 4 Vital Pillars

The 4 Vital Pillars of HPC Infrastructure:

Phase 2: Deploying the Mini-Cluster with Docker Compose

Key Configuration Highlights

1. In slurm.conf:

2. In docker-compose.yml:

Phase 3: Database Accounting & Multi-Tenancy (sacctmgr)

Initializing Accounting via sacctmgr:

Enforcing Resource Limits & QoS Quotas:

Phase 4: GPU Cluster Simulation via Generic Resources (GRES)

How It Works:

Phase 5: Turning Your Host Machine into a Native Login Node

Phase 6: Slurm CLI Toolkit & Workflow Mastery

1. sinfo — Inspecting Nodes & Partitions

2. squeue & scancel — Active Queue & Job Control

3. sbatch, srun & salloc — Job Execution (Batch & Interactive)

A. Asynchronous Batch Execution (sbatch):

B. Synchronous & Interactive Debugging (srun):

C. Interactive Resource Reservation (salloc):

4. sacct, sstat & sreport — Accounting, Profiling & Auditing

5. scontrol & sacctmgr — Cluster Administration & Multi-Tenancy

Phase 7: Battle-Tested Troubleshooting (War Stories)

Conclusion

Published

Category

Tags

Contact

1. In `slurm.conf`:

2. In `docker-compose.yml`:

Phase 3: Database Accounting & Multi-Tenancy (`sacctmgr`)

Initializing Accounting via `sacctmgr`:

1. `sinfo` — Inspecting Nodes & Partitions

2. `squeue` & `scancel` — Active Queue & Job Control

3. `sbatch`, `srun` & `salloc` — Job Execution (Batch & Interactive)

A. Asynchronous Batch Execution (`sbatch`):

B. Synchronous & Interactive Debugging (`srun`):

C. Interactive Resource Reservation (`salloc`):

4. `sacct`, `sstat` & `sreport` — Accounting, Profiling & Auditing

5. `scontrol` & `sacctmgr` — Cluster Administration & Multi-Tenancy