Exercise Validation Details¶
Captured from slurm-training-val at 2026-10-19T01:05:05Z. See the
summary page for the full toolchain and
run-level context.
Each entry below shows the command(s) that the validation harness ran for that exercise and the real output that came back. A non-zero exit code is flagged with a comment line inside the code block — some exercises intentionally trigger a failure (e.g., resource over-request demonstrations).
admin/accounts-fairshare¶
Exercise 1¶
Captured 2026-10-19T01:05:20Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sudo /opt/slurm/bin/sacctmgr -i add account val_research Description='Research Division'
Adding Account(s)
val_research
Settings
Description = research division
Organization = Parent/Account Name
Associations =
C = slurm-training-val A = val_research
Settings
$ sudo /opt/slurm/bin/sacctmgr -i add account val_structural_biology parent=val_research Description='Structural Biology Dept'
Adding Account(s)
val_structural_biology
Settings
Description = structural biology dept
Organization = Parent/Account Name
Associations =
C = slurm-training-val A = val_structural_biology
Settings
Parent = val_research
$ sudo /opt/slurm/bin/sacctmgr -i add account val_cryo_lab parent=val_structural_biology Description='Cryo-EM Lab'
Adding Account(s)
val_cryo_lab
Settings
Description = cryo-em lab
Organization = Parent/Account Name
Associations =
C = slurm-training-val A = val_cryo_lab
Settings
Parent = val_structural_biology
$ sudo /opt/slurm/bin/sacctmgr -i add user val_pi_chen account=val_cryo_lab fairshare=20 qos=normal defaultqos=normal
Adding User(s)
val_pi_chen
Settings
Associations =
C = slurm-training-val A = val_cryo_lab U = val_pi_chen
Non Default Settings
Fairshare = 20
QOS = normal
DefQOS = normal
$ sudo /opt/slurm/bin/sacctmgr -i add user val_postdoc_kim account=val_cryo_lab fairshare=10 qos=normal defaultqos=normal
Adding User(s)
val_postdoc_kim
Settings
Associations =
C = slurm-training-val A = val_cryo_lab U = val_postdoc_kim
Non Default Settings
Fairshare = 10
QOS = normal
DefQOS = normal
$ sacctmgr show association where account=val_cryo_lab format=Account,User,Fairshare tree withd
Account User Share
-------------------- ---------- ---------
val_cryo_lab 1
val_cryo_lab val_pi_ch+ 20
val_cryo_lab val_postd+ 10
$ sacctmgr show user val_pi_chen,val_postdoc_kim format=User%-20,DefaultAccount%-20
User Def Acct
-------------------- --------------------
val_pi_chen val_cryo_lab
val_postdoc_kim val_cryo_lab
Exercise 2¶
Captured 2026-10-19T01:05:21Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sshare -l | head -20
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 812 1.000000 cpu=0,mem=0,energy=0,node=0,b+
pcdefault 1 0.006579 794 0.977897 0.977897 0.006728 cpu=0,mem=0,energy=0,node=0,b+
pcdefault ec2-user 1 0.500000 794 0.977897 1.000000 0.250000 0.500000 cpu=0,mem=0,energy=0,node=0,b+
research 50 0.328947 17 0.022103 0.022103 14.882605 cpu=0,mem=0,energy=0,node=0,b+
training 100 0.657895 0 0.000000 0.000000 inf cpu=0,mem=0,energy=0,node=0,b+
Exercise 3¶
Captured 2026-10-19T01:05:24Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sudo /opt/slurm/bin/sacctmgr -i modify account val_structural_biology set fairshare=40
Modified account associations...
C = slurm-training-val A = val_structural_biology of val_research
$ sudo /opt/slurm/bin/sacctmgr -i modify account val_chemistry set fairshare=60
Modified account associations...
C = slurm-training-val A = val_chemistry of val_research
$ sacctmgr show account val_structural_biology,val_chemistry format=Account%-25,Description%-25,Organization%-25
Account Descr Org
------------------------- ------------------------- -------------------------
val_chemistry val_chemistry val_research
val_structural_biology val_structural_biology val_research
Exercise 4¶
Captured 2026-10-19T01:05:24Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sreport cluster AccountUtilizationByUser start=now-7days --tres=cpu | head -15
--------------------------------------------------------------------------------
Cluster/Account/User Utilization 2026-10-12T01:00:00 - 2026-10-19T00:59:59 (604800 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
Cluster Account Login Proper Name TRES Name Used
--------- --------------- --------- --------------- -------------- --------
slurm-tr+ root cpu 14
slurm-tr+ pcdefault cpu 13
slurm-tr+ pcdefault ec2-user AWS ParallelCl+ cpu 13
slurm-tr+ research cpu 0
slurm-tr+ research testuser1 cpu 0
$ sreport user TopUsage start=now-7days --tres=cpu TopCount=10 | head -15
--------------------------------------------------------------------------------
Top 10 Users 2026-10-12T01:00:00 - 2026-10-19T00:59:59 (604800 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
Cluster Login Proper Name Account TRES Name Used
--------- --------- --------------- --------------- -------------- --------
slurm-tr+ ec2-user AWS ParallelCl+ pcdefault cpu 13
slurm-tr+ testuser1 research cpu 0
admin/configuration¶
Exercise 1¶
Captured 2026-10-19T01:05:26Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 2¶
Captured 2026-10-19T01:05:26Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sudo /opt/slurm/bin/scontrol setdebug debug2
$ scontrol show config | grep SlurmctldDebug
SlurmctldDebug = debug2
$ scontrol show config | grep SlurmctldDebug
SlurmctldDebug = info
Exercise 3¶
Captured 2026-10-19T01:05:27Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol show partition val_extra
PartitionName=val_extra
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=NO QoS=N/A
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
Nodes=val_node1
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=1 TotalNodes=1 SelectTypeParameters=NONE
JobDefaults=(null)
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
TRES=cpu=1,mem=128M,node=1,billing=1
ResumeTimeout=GLOBAL SuspendTimeout=GLOBAL SuspendTime=GLOBAL PowerDownOnIdle=NO
$ scontrol show node val_node1 || true
Node val_node1 not found
$ sudo grep '^NodeName=val_node1' /opt/slurm/etc/slurm.conf
NodeName=val_node1 CPUs=1 RealMemory=128 State=FUTURE
Exercise 4¶
Captured 2026-10-19T01:05:28Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol show partition val_extra2
PartitionName=val_extra2
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=NO QoS=N/A
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
Nodes=val_node2
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=2 TotalNodes=1 SelectTypeParameters=NONE
JobDefaults=(null)
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
TRES=cpu=2,mem=256M,node=1,billing=2
ResumeTimeout=GLOBAL SuspendTimeout=GLOBAL SuspendTime=GLOBAL PowerDownOnIdle=NO
admin/high-availability¶
Exercise 1¶
Captured 2026-10-19T01:05:29Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol show config | grep SlurmctldHost
SlurmctldHost[0] = ip-172-31-84-168(172.31.84.168)
$ scontrol show config | grep StateSaveLocation
StateSaveLocation = /var/spool/slurm.state
Exercise 2¶
Captured 2026-10-19T01:05:29Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol ping
Slurmctld(primary) at ip-172-31-84-168 is UP
$ sinfo --summarize
PARTITION AVAIL TIMELIMIT NODES(A/I/O/T) NODELIST
batch* up infinite 0/4/0/4 batch-dy-compute-[1-4]
debug up infinite 0/2/0/2 debug-dy-small-[1-2]
gpu up infinite 0/1/0/1 gpu-dy-t4-1
highmem up infinite 0/1/0/1 highmem-dy-large-1
admin/installation¶
Exercise 1¶
Captured 2026-10-19T01:05:30Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ munge -n | unmunge | head -5
STATUS: Success (0)
ENCODE_HOST: ip-172-31-84-168.us-east-2.compute.internal (172.31.84.168)
ENCODE_TIME: 2026-10-19 01:05:30 +0000 (1792371930)
DECODE_TIME: 2026-10-19 01:05:30 +0000 (1792371930)
TTL: 300
Exercise 2¶
Captured 2026-10-19T01:05:30Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ systemctl is-active slurmdbd
active
$ sacctmgr -nP show cluster format=Cluster,ControlHost,ControlPort,RPC
slurm-training-val|172.31.84.168|6820|11264
Exercise 3¶
Captured 2026-10-19T01:05:30Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacctmgr show cluster format=Cluster,ControlHost,ControlPort,RPC
Cluster ControlHost ControlPort RPC
---------- --------------- ------------ -----
slurm-tra+ 172.31.84.168 6820 11264
$ sacctmgr show account format=Account,Description,Organization
Account Descr Org
---------- -------------------- --------------------
pcdefault parallelcluster def+ none
research research bioteam
root default root account root
training training bioteam
$ sacctmgr show association format=Cluster,Account,User,Fairshare | head -15
Cluster Account User Share
---------- ---------- ---------- ---------
slurm-tra+ root 1
slurm-tra+ root root 1
slurm-tra+ pcdefault 1
slurm-tra+ pcdefault ec2-user 1
slurm-tra+ pcdefault slurm 1
slurm-tra+ research 50
slurm-tra+ research testuser1 1
slurm-tra+ training 100
admin/maintenance-operations¶
Exercise 1¶
Captured 2026-10-19T01:05:31Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sudo /opt/slurm/bin/scontrol update NodeName=debug-dy-small-1 State=DRAIN Reason='val: maintenance test'
$ scontrol show node debug-dy-small-1 | grep -E '(State|Reason)'
State=IDLE+CLOUD+DRAIN+POWERED_DOWN+NOT_RESPONDING ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
Reason=val: maintenance test [root@2026-10-19T01:05:32]
$ sudo /opt/slurm/bin/scontrol update NodeName=debug-dy-small-1 State=RESUME
$ scontrol show node debug-dy-small-1 | grep State=
State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
Exercise 2¶
Captured 2026-10-19T01:05:32Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sinfo -R
REASON USER TIMESTAMP NODELIST
$ sinfo -o '%T %D' --noheader | sort | uniq -c | sort -rn
1 idle~ 8
$ sinfo -o '%T %C' --noheader | head -10
idle~ 0/48/0/48
Exercise 3¶
Captured 2026-10-19T01:05:32Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ awk '$3 ~ /^(nfs|nfs4|cifs|smb3|lustre|beegfs|gpfs)$/ {print $3, $2}' /proc/mounts | sort -k2
$ awk '$3 ~ /^(nfs|nfs4|cifs|smb3|lustre|beegfs|gpfs)$/ {print $2}' /proc/mounts | sort -u | while read -r m; do echo "== $m"; timeout 20 find "$m" -maxdepth 1 -newer /etc/hostname | head; [ "${PIPESTATUS[0]}" -eq 124 ] && echo "probe timed out"; done; true
admin/monitoring-accounting¶
Exercise 1¶
Captured 2026-10-19T01:05:33Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sreport cluster utilization start=now-7days
--------------------------------------------------------------------------------
Cluster Utilization 2026-10-12T01:00:00 - 2026-10-19T00:59:59
Usage reported in CPU Minutes
--------------------------------------------------------------------------------
Cluster Allocate Down PLND Dow Idle Planned Reported
--------- -------- -------- -------- -------- -------- --------
slurm-tr+ 14 0 282356 201437 34 483840
Exercise 2¶
Captured 2026-10-19T01:05:33Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sreport user TopUsage start=now-7days --tres=cpu TopCount=5
--------------------------------------------------------------------------------
Top 5 Users 2026-10-12T01:00:00 - 2026-10-19T00:59:59 (604800 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
Cluster Login Proper Name Account TRES Name Used
--------- --------- --------------- --------------- -------------- --------
slurm-tr+ ec2-user AWS ParallelCl+ pcdefault cpu 13
slurm-tr+ testuser1 research cpu 0
Exercise 3¶
Captured 2026-10-19T01:05:33Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sdiag
*******************************************************
sdiag output at Mon Oct 19 01:05:33 2026 (1792371933)
Data since Mon Oct 19 01:05:29 2026 (1792371929)
*******************************************************
Server thread count: 1
RPC queue enabled: 0
Agent queue size: 0
Agent count: 0
Agent thread count: 0
DBD Agent queue size: 0
Jobs submitted: 0
Jobs started: 0
Jobs completed: 0
Jobs canceled: 0
Jobs failed: 0
Job states ts: Mon Oct 19 01:05:29 2026 (1792371929)
Jobs pending: 0
Jobs running: 0
Main schedule statistics (microseconds):
Last cycle: 17
Max cycle: 17
Total cycles: 1
Mean cycle: 17
Mean depth cycle: 0
Last queue length: 0
Main scheduler exit:
End of job queue: 1
Hit default_queue_depth: 0
Hit sched_max_job_start: 0
Blocked on licenses: 0
Hit max_rpc_cnt: 0
Timeout (max_sched_time): 0
Backfilling stats
Total backfilled jobs (since last slurm start): 0
Total backfilled jobs (since last stats cycle start): 0
Total backfilled heterogeneous job components: 0
Total cycles: 0
Last cycle when: Mon Oct 19 00:51:48 2026 (1792371108)
Last cycle: 0
Max cycle: 0
Last depth cycle: 0
Last depth cycle (try sched): 0
Last queue length: 0
Last table size: 0
Backfill exit
End of job queue: 0
Hit bf_max_job_start: 0
Hit bf_max_job_test: 0
System state changed: 0
Hit table size limit (bf_node_space_size): 0
Timeout (bf_max_time): 0
Latency for 1000 calls to gettimeofday(): 27 microseconds
Remote Procedure Call statistics by message type
REQUEST_PARTITION_INFO ( 2009) count:6 ave_time:58 total_time:352
REQUEST_NODE_INFO ( 2007) count:6 ave_time:158 total_time:949
REQUEST_UPDATE_NODE ( 3002) count:5 ave_time:147 total_time:738
REQUEST_PING ( 1008) count:3 ave_time:60 total_time:182
REQUEST_BUILD_INFO ( 2001) count:2 ave_time:151 total_time:303
Remote Procedure Call statistics by user
ec2-user ( 1000) count:17 ave_time:105 total_time:1786
root ( 0) count:5 ave_time:147 total_time:738
Pending RPC statistics
No pending RPCs
Exercise 4¶
Captured 2026-10-19T01:05:33Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct --starttime=now-7days --state=OUT_OF_MEMORY --format=JobID,User,JobName,ReqMem,MaxRSS,Elapsed,State,ExitCode
JobID User JobName ReqMem MaxRSS Elapsed State ExitCode
------------ --------- ---------- ---------- ---------- ---------- ---------- --------
$ sacct --starttime=now-7days --state=COMPLETED --format=JobID%-12,User%-10,JobName%-20,ReqMem,MaxRSS,Elapsed | head -10
JobID User JobName ReqMem MaxRSS Elapsed
------------ ---------- -------------------- ---------- ---------- ----------
Exercise 6¶
Captured 2026-10-19T01:05:34Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct -a -X -n -P -s RUNNING -S 2000-01-01 -o JobIDRaw,Start,State | wc -l
0
$ squeue -h -t R | wc -l
0
$ comm -23 <(sacct -a -X -n -P -s RUNNING -S 2000-01-01 -o JobIDRaw | sort) <(squeue -h -t R -o %i | sort)
Exercise 7¶
Captured 2026-10-19T01:08:44Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sudo grep -E 'JobId=42( |$)|JobId=42_' /var/log/slurmctld.log
[2026-10-19T01:05:34.308] _slurm_rpc_submit_batch_job: JobId=42 SLUID=sBHSM58V2H3400 InitPrio=3500 usec=490
[2026-10-19T01:05:35.007] sched: Allocate JobId=42 NodeList=batch-dy-compute-1 #CPUs=1 Partition=batch
[2026-10-19T01:06:01.027] POWER: no more nodes to resume for job JobId=42
[2026-10-19T01:08:29.007] job_time_limit: Configuration for JobId=42 complete
[2026-10-19T01:08:29.007] Resetting JobId=42 start time for node power up
[2026-10-19T01:08:34.358] _job_complete: JobId=42 WEXITSTATUS 0
[2026-10-19T01:08:34.358] _job_complete: JobId=42 done
admin/partitions-qos¶
Exercise 1¶
Captured 2026-10-19T01:08:45Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol show partition val_long
PartitionName=val_long
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=NO QoS=N/A
DefaultTime=04:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=14-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
Nodes=debug-dy-small-[1-2]
PriorityJobFactor=1 PriorityTier=50 RootOnly=NO ReqResv=NO OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=4 TotalNodes=2 SelectTypeParameters=NONE
JobDefaults=(null)
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
TRES=cpu=4,mem=7782M,node=2,billing=4
ResumeTimeout=GLOBAL SuspendTimeout=GLOBAL SuspendTime=GLOBAL PowerDownOnIdle=NO
$ sinfo -p val_long
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
val_long up 14-00:00:0 2 idle~ debug-dy-small-[1-2]
Exercise 2¶
Captured 2026-10-19T01:08:46Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
# exit=1
$ sudo /opt/slurm/bin/sacctmgr -i add qos val_pq_restricted set MaxTRESPerUser=cpu=32,gres/gpu=2 MaxJobsPerUser=10 Priority=0
Nothing new added.
$ sacctmgr show qos val_pq_restricted format=Name%-20,MaxTRESPerUser%-30,MaxJobsPerUser,Priority
Name MaxTRESPU MaxJobsPU Priority
-------------------- ------------------------------ --------- ----------
val_pq_restricted cpu=32,gres/gpu=2 10 0
Exercise 3¶
Captured 2026-10-19T01:08:47Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sudo /opt/slurm/bin/sacctmgr -i modify account training set qos=normal,val_pq_restricted defaultqos=normal
Modified account associations...
C = slurm-training-val A = training of root
$ sacctmgr show association where account=training format=Account,User,QOS,DefaultQOS
Account User QOS Def QOS
---------- ---------- -------------------- ---------
training normal,val_pq_restr+ normal
Exercise 4¶
Captured 2026-10-19T01:09:03Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ squeue -u testuser1 -o '%.8i %.2t %.30R'
JOBID ST NODELIST(REASON)
104 CF batch-dy-compute-2
105 PD (QOSMaxCpuPerUserLimit)
103 R batch-dy-compute-1
Exercise 5¶
Captured 2026-10-19T01:09:04Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacctmgr show qos format=name,priority,maxwall,grptres
Name Priority MaxWall GrpTRES
---------- ---------- ----------- -------------
normal 0
high 100
low 10
$ scontrol show config | grep -E 'PriorityType|PriorityWeight'
PriorityType = priority/multifactor
PriorityWeightAge = 1000
PriorityWeightAssoc = 0
PriorityWeightFairShare = 10000
PriorityWeightJobSize = 0
PriorityWeightPartition = 1000
PriorityWeightQOS = 10000
PriorityWeightTRES = (null)
$ sshare -a -l 2>&1 | head -20
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 834 1.000000 cpu=0,mem=0,energy=0,node=0,b+
root root 1 0.006579 0 0.000000 0.000000 1.000000 inf cpu=0,mem=0,energy=0,node=0,b+
pcdefault 1 0.006579 799 0.958109 0.958109 0.006867 cpu=0,mem=0,energy=0,node=0,b+
pcdefault ec2-user 1 0.500000 799 0.958109 1.000000 0.250000 0.500000 cpu=0,mem=0,energy=0,node=0,b+
pcdefault slurm 1 0.500000 0 0.000000 0.000000 0.500000 inf cpu=0,mem=0,energy=0,node=0,b+
research 50 0.328947 34 0.041891 0.041891 7.852417 cpu=0,mem=0,energy=0,node=0,b+
research testuser1 1 1.000000 34 0.041891 1.000000 0.750000 1.000000 cpu=0,mem=0,energy=0,node=0,b+
training 100 0.657895 0 0.000000 0.000000 inf cpu=0,mem=0,energy=0,node=0,b+
$ scontrol -a show partition | grep -E 'PartitionName|Hidden|AllowGroups|AllowQos|MaxTime'
PartitionName=batch
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
PartitionName=debug
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
PartitionName=gpu
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
PartitionName=highmem
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
$ sacctmgr show assoc format=user,account,qos,defaultqos
User Account QOS Def QOS
---------- ---------- -------------------- ---------
root normal
root root normal
pcdefault normal
ec2-user pcdefault normal
slurm pcdefault normal
research normal
testuser1 research normal normal
training normal normal
$ scontrol show config | grep -E 'SchedulerType|PriorityType|PreemptMode|SelectTypeParameters'
PreemptMode = OFF
PriorityType = priority/multifactor
SchedulerType = sched/backfill
SelectTypeParameters = CR_CPU_MEMORY
admin/policies-priority¶
Exercise 1¶
Captured 2026-10-19T01:09:08Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sprio -l
JOBID PARTITION USER ACCOUNT PRIORITY SITE AGE ASSOC FAIRSHARE JOBSIZE PARTITION QOSNAME QOS NICE TRES
$ sprio -n -l
JOBID PARTITION USER ACCOUNT PRIORITY AGE ASSOC FAIRSHARE JOBSIZE PARTITION QOSNAME QOS TRES
$ sshare -a -l | head -10
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 834 1.000000 cpu=0,mem=0,energy=0,node=0,b+
root root 1 0.006579 0 0.000000 0.000000 1.000000 inf cpu=0,mem=0,energy=0,node=0,b+
pcdefault 1 0.006579 799 0.958109 0.958109 0.006867 cpu=0,mem=0,energy=0,node=0,b+
pcdefault ec2-user 1 0.500000 799 0.958109 1.000000 0.250000 0.500000 cpu=0,mem=0,energy=0,node=0,b+
pcdefault slurm 1 0.500000 0 0.000000 0.000000 0.500000 inf cpu=0,mem=0,energy=0,node=0,b+
research 50 0.328947 34 0.041891 0.041891 7.852417 cpu=0,mem=0,energy=0,node=0,b+
research testuser1 1 1.000000 34 0.041891 1.000000 0.750000 1.000000 cpu=0,mem=0,energy=0,node=0,b+
training 100 0.657895 0 0.000000 0.000000 inf cpu=0,mem=0,energy=0,node=0,b+
Exercise 2¶
Captured 2026-10-19T01:09:08Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sudo /opt/slurm/bin/scontrol create reservation ReservationName=val_pp_resv StartTime=now+3600 Duration=00:30:00 Nodes=debug-dy-small-1 Users=ec2-user Flags=MAINT,IGNORE_JOBS
Reservation created: val_pp_resv
$ scontrol show reservation val_pp_resv
ReservationName=val_pp_resv StartTime=2026-10-19T02:09:08 EndTime=2026-10-19T02:39:08 Duration=00:30:00
Nodes=debug-dy-small-1 NodeCnt=1 CoreCnt=2 Features=(null) PartitionName=(null) Flags=MAINT,IGNORE_JOBS,SPEC_NODES
TRES=cpu=2
AllowedPartitions=(null) QOS=(null)
Users=ec2-user Groups=(null) Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null)
MaxStartDelay=(null)
$ sudo /opt/slurm/bin/scontrol delete ReservationName=val_pp_resv
Exercise 3¶
Captured 2026-10-19T01:09:08Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sdiag | head -40
*******************************************************
sdiag output at Mon Oct 19 01:09:08 2026 (1792372148)
Data since Mon Oct 19 01:09:05 2026 (1792372145)
*******************************************************
Server thread count: 1
RPC queue enabled: 0
Agent queue size: 0
Agent count: 0
Agent thread count: 0
DBD Agent queue size: 0
Jobs submitted: 1
Jobs started: 0
Jobs completed: 0
Jobs canceled: 1
Jobs failed: 0
Job states ts: Mon Oct 19 01:09:05 2026 (1792372145)
Jobs pending: 0
Jobs running: 0
Main schedule statistics (microseconds):
Last cycle: 11
Max cycle: 12
Total cycles: 2
Mean cycle: 11
Mean depth cycle: 0
Last queue length: 0
Main scheduler exit:
End of job queue: 2
Hit default_queue_depth: 0
Hit sched_max_job_start: 0
Blocked on licenses: 0
Hit max_rpc_cnt: 0
Timeout (max_sched_time): 0
Backfilling stats
Total backfilled jobs (since last slurm start): 0
Total backfilled jobs (since last stats cycle start): 0
admin/resource-management¶
Exercise 1¶
Captured 2026-10-19T01:09:22Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ echo 'NodeName=batch-dy-compute-1 CPUs=8 RealMemory=31129 State=CLOUD Feature=dynamic,t3.2xlarge,compute Weight=1000 Gres=val_fpga:2'
NodeName=batch-dy-compute-1 CPUs=8 RealMemory=31129 State=CLOUD Feature=dynamic,t3.2xlarge,compute Weight=1000 Gres=val_fpga:2
$ echo 'NodeName=batch-dy-compute-1 Name=val_fpga Count=2'
NodeName=batch-dy-compute-1 Name=val_fpga Count=2
$ ssh batch-dy-compute-1 'sudo systemctl restart slurmd'
Warning: Permanently added 'batch-dy-compute-1' (ED25519) to the list of known hosts.
$ grep val_fpga /opt/slurm/etc/pcluster/slurm_parallelcluster_batch_partition.conf /opt/slurm/etc/pcluster/slurm_parallelcluster_batch_gres.conf
/opt/slurm/etc/pcluster/slurm_parallelcluster_batch_gres.conf:NodeName=batch-dy-compute-1 Name=val_fpga Count=2
$ scontrol show node batch-dy-compute-1 | grep -E '(NodeName|Gres|CfgTRES)'
NodeName=batch-dy-compute-1 Arch=x86_64 CoresPerSocket=1
Gres=(null)
CfgTRES=cpu=8,mem=31129M,billing=8
Exercise 2¶
Captured 2026-10-19T01:09:30Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol show node gpu-dy-t4-1 | grep -E '(NodeName|Gres|CfgTRES|AllocTRES)'
NodeName=gpu-dy-t4-1 CoresPerSocket=1
Gres=gpu:t4:1
CfgTRES=cpu=4,mem=15564M,billing=4,gres/gpu=1
AllocTRES=
Exercise 3¶
Captured 2026-10-19T01:09:31Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol show licenses
LicenseName=val_glide
Total=5 Used=0 Free=5 Reserved=0 Remote=no
$ scontrol show licenses
LicenseName=val_glide
Total=5 Used=0 Free=5 Reserved=0 Remote=no
Exercise 4¶
Captured 2026-10-19T01:09:34Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ cat /opt/slurm/etc/cgroup.conf 2>/dev/null || echo '(cgroup.conf not at /opt/slurm/etc/)'
###
# Slurm cgroup support configuration file
###
ConstrainCores=yes
#
# WARNING!!! The slurm_parallelcluster_cgroup.conf file included below can be updated by the pcluster process.
# Please do not edit it.
include slurm_parallelcluster_cgroup.conf
$ sbatch --mem=100M --time=00:05:00 --output=/dev/null --job-name=val_oom --wrap='/shared/python/3.12/bin/python3 -c "x = bytearray(500 * 1024 * 1024); import time; time.sleep(60)"'
Submitted batch job 42
$ sacct -j 42 --format=JobID,JobName%-12,State%-20,ExitCode,MaxRSS
JobID JobName State ExitCode MaxRSS
------------ ------------ -------------------- -------- ----------
42 val_oom OUT_OF_MEMORY 0:125
42.batch batch OUT_OF_MEMORY 0:125 102196K
Exercise 5¶
Captured 2026-10-19T01:14:38Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol show nodes | grep -E '(NodeName|AvailableFeatures)' | head -20
NodeName=batch-dy-compute-1 Arch=x86_64 CoresPerSocket=1
AvailableFeatures=dynamic,t3.2xlarge,compute
NodeName=batch-dy-compute-2 CoresPerSocket=1
AvailableFeatures=dynamic,t3.2xlarge,compute
NodeName=batch-dy-compute-3 CoresPerSocket=1
AvailableFeatures=dynamic,t3.2xlarge,compute
NodeName=batch-dy-compute-4 CoresPerSocket=1
AvailableFeatures=dynamic,t3.2xlarge,compute
NodeName=debug-dy-small-1 CoresPerSocket=1
AvailableFeatures=dynamic,t3.medium,small
NodeName=debug-dy-small-2 CoresPerSocket=1
AvailableFeatures=dynamic,t3.medium,small
NodeName=gpu-dy-t4-1 CoresPerSocket=1
AvailableFeatures=dynamic,g4dn.xlarge,t4,gpu
NodeName=highmem-dy-large-1 CoresPerSocket=1
AvailableFeatures=dynamic,t3.2xlarge,large
Exercise 6¶
Captured 2026-10-19T01:14:39Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol show config | grep -E 'SelectTypeParameters|DefMemPerNode|TaskPlugin|ProctrackType|JobAcctGatherType'
DefMemPerNode = UNLIMITED
JobAcctGatherType = jobacct_gather/cgroup
ProctrackType = proctrack/cgroup
SelectTypeParameters = CR_CPU_MEMORY
TaskPlugin = task/cgroup,task/affinity
TaskPluginParam = (null type)
Exercise 7¶
Captured 2026-10-19T01:14:40Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ srun -J val_rm_e7 --chdir=/tmp -c 2 bash -c 'nproc; grep Cpus_allowed_list /proc/self/status; echo SLURM_CPUS_ON_NODE=$SLURM_CPUS_ON_NODE'
2
Cpus_allowed_list: 0,42
SLURM_CPUS_ON_NODE=2
$ srun -J val_rm_e7 --chdir=/tmp -c 2 python3 -c 'import os; print(os.cpu_count(), len(os.sched_getaffinity(0)))'
8 2
Exercise 8¶
Captured 2026-10-19T01:14:42Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacctmgr show tres
Type Name ID
-------- --------------- ------
cpu 1
mem 2
energy 3
node 4
billing 5
fs disk 6
vmem 7
pages 8
gres gpu 1001
gres gpumem 1002
gres gpuutil 1003
admin/slurm-architecture¶
Exercise 1¶
Captured 2026-10-19T01:14:43Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ systemctl is-active slurmctld slurmdbd
active
active
$ ps -eo comm,pid | grep -E 'slurm(ctld|dbd|d|restd)' | head -10
slurmdbd 4038
slurmctld 4093
Exercise 2¶
Captured 2026-10-19T01:14:44Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ slurmctld -V
slurm 25.11.8
$ scontrol show config | grep SLURM_VERSION
SLURM_VERSION = 25.11.8
$ scontrol ping
Slurmctld(primary) at ip-172-31-84-168 is UP
admin/troubleshooting¶
Exercise 1¶
Captured 2026-10-19T01:14:47Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ squeue -j 42 -o '%.8i %.9P %.20j %.2t %.30R'
JOBID PARTITION NAME ST NODELIST(REASON)
42 batch val_ts1 PD (JobHeldUser)
$ scontrol show job 42 | grep -E '(JobState|Reason)'
JobState=PENDING Reason=JobHeldUser Dependency=(null)
Exercise 2¶
Captured 2026-10-19T01:14:47Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sudo /opt/slurm/bin/scontrol update NodeName=debug-dy-small-2 State=DRAIN Reason='val: ts2 drain test'
$ sinfo -n debug-dy-small-2
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up infinite 0 n/a
debug up infinite 1 drain~ debug-dy-small-2
gpu up infinite 0 n/a
highmem up infinite 0 n/a
$ scontrol show node debug-dy-small-2 | grep -E '(State|Reason)'
State=IDLE+CLOUD+DRAIN+POWERED_DOWN ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
Reason=val: ts2 drain test [root@2026-10-19T01:14:47]
Exercise 3¶
Captured 2026-10-19T01:14:48Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sudo /opt/slurm/bin/scontrol update NodeName=debug-dy-small-2 State=RESUME
$ scontrol show node debug-dy-small-2 | grep State=
State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
Exercise 4¶
Captured 2026-10-19T01:14:50Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct --starttime=now-7days --state=OUT_OF_MEMORY --format=JobID%-12,User%-10,JobName%-20,ReqMem,MaxRSS,Elapsed,ExitCode,NodeList
JobID User JobName ReqMem MaxRSS Elapsed ExitCode NodeList
------------ ---------- -------------------- ---------- ---------- ---------- -------- ---------------
$ sacct --starttime=now-7days --format=JobID,User,State,ExitCode | grep '0:9' | head -10
$ sacct --starttime=now-7days --format=JobID,User,State,ExitCode | grep '0:9' | head -10 || echo '(no SIGKILL exits in window)'
Exercise 5¶
Captured 2026-10-19T01:14:51Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sinfo -N -l | head -20
Mon Oct 19 01:14:51 2026
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
batch-dy-compute-1 1 batch* idle 8 8:1:1 31129 0 1000 dynamic, none
batch-dy-compute-2 1 batch* idle~ 8 8:1:1 31129 0 1000 dynamic, none
batch-dy-compute-3 1 batch* idle~ 8 8:1:1 31129 0 1000 dynamic, none
batch-dy-compute-4 1 batch* idle~ 8 8:1:1 31129 0 1000 dynamic, none
debug-dy-small-1 1 debug idle~ 2 2:1:1 3891 0 1000 dynamic, none
debug-dy-small-2 1 debug idle~ 2 2:1:1 3891 0 1000 dynamic, none
gpu-dy-t4-1 1 gpu idle~ 4 4:1:1 15564 0 1000 dynamic, none
highmem-dy-large-1 1 highmem idle~ 8 8:1:1 31129 0 1000 dynamic, none
$ sinfo -R --noheader || echo '(no nodes with REASON set)'
Exercise 6¶
Captured 2026-10-19T01:14:56Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol show job 113 | grep -E 'JobState|Reason|ExitCode|RunTime'
JobState=FAILED Reason=RaisedSignal:53(Real-time_signal_19) Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:53
RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A
Exercise 7¶
Captured 2026-10-19T01:14:56Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ cd /var/log 2>/dev/null; sudo zgrep -hE 'error|fatal' slurmctld.log* 2>/dev/null | sed -E 's/[0-9]+/N/g' | sort | uniq -c | sort -rn | head -20
45 [N-N-NTN:N:N.N] error: chdir(/var/log): Permission denied
45 [N-N-NTN:N:N.N] error: Configured MailProg is invalid
8 [N-N-NTN:N:N.N] error: gres_node_state_unpack: no plugin configured to unpack data type N from node batch-dy-compute-N
4 [N-N-NTN:N:N.N] error: gres_node_state_unpack: no plugin configured to unpack data type N from node debug-dy-small-N
4 [N-N-NTN:N:N.N] error: _find_node_record: lookup failure for node "val_nodeN"
4 [N-N-NTN:N:N.N] error: Node val_nodeN has vanished from configuration
2 [N-N-NTN:N:N.N] slurmscriptd: error: run_command: MailProg can not be executed (/bin/mail) No such file or directory
2 [N-N-NTN:N:N.N] slurmscriptd: error: _run_script: JobId=N MailProg exit status N:N
2 [N-N-NTN:N:N.N] error: gres_node_state_unpack: no plugin configured to unpack data type N from node highmem-dy-large-N
2 [N-N-NTN:N:N.N] error: gres_node_state_unpack: no plugin configured to unpack data type N from node gpu-dy-tN-N
2 [N-N-NTN:N:N.N] error: MailProg returned error, it's output was 'Run command failed - configuration error'
1 [N-N-NTN:N:N.N] error: Nodes batch-dy-compute-[N-N] not responding
Exercise 8¶
Captured 2026-10-19T01:14:57Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct -a -X -S now-7days -o JobID,State,ExitCode,ElapsedRaw -P -n | awk -F'|' '$2!~/COMPLETED|RUNNING|PENDING/ { b = ($4<60)?"<1m":($4<600)?"1-10m":($4<3600)?"10-60m":">1h"; c[$3" "b]++ } END { for (k in c) print c[k], k }' | sort -rn | head -20
20 0:0 <1m
8 0:15 <1m
3 1:0 <1m
2 0:53 <1m
2 0:125 <1m
deployment/aws-troubleshooting¶
Exercise 1¶
Captured 2026-10-19T01:14:58Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sinfo -R
REASON USER TIMESTAMP NODELIST
$ scontrol show node batch-dy-compute-1 | grep -E '(Reason|State)='
State=IDLE+CLOUD ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
Exercise 2¶
Captured 2026-10-19T01:14:58Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up infinite 3 idle~ batch-dy-compute-[2-4]
batch* up infinite 1 idle batch-dy-compute-1
debug up infinite 2 idle~ debug-dy-small-[1-2]
gpu up infinite 1 idle~ gpu-dy-t4-1
highmem up infinite 1 idle~ highmem-dy-large-1
$ sudo grep 'Node bootstrap error' /var/log/parallelcluster/clustermgtd 2>/dev/null | tail -5 || echo '(no bootstrap errors found)'
$ sudo tail -3 /var/log/parallelcluster/clustermgtd
2026-10-19 01:14:41,42 - [slurm_plugin.clustermgtd:_maintain_nodes] - INFO - Performing node maintenance actions
2026-10-19 01:14:41,43 - [slurm_plugin.clustermgtd:_maintain_nodes] - INFO - Following nodes are currently in replacement: (x0) []
2026-10-19 01:14:41,43 - [slurm_plugin.clustermgtd:_terminate_orphaned_instances] - INFO - Checking for orphaned instance
user/best-practices¶
Exercise 1¶
Captured 2026-10-19T01:15:35Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct -j 42 --format=JobID,JobName,Elapsed,MaxRSS,MaxVMSize,CPUTime,ReqMem,ReqCPUS,State
JobID JobName Elapsed MaxRSS MaxVMSize CPUTime ReqMem ReqCPUS State
------------ ---------- ---------- ---------- ---------- ---------- ---------- -------- ----------
42 val_bp1 00:00:30 00:02:00 8G 4 COMPLETED
42.batch batch 00:00:30 952K 0 00:02:00 4 COMPLETED
Exercise 2¶
Captured 2026-10-19T01:18:02Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct -j 42,43 --format=JobID,JobName,Elapsed,MaxRSS,ReqMem,ReqCPUS,CPUTime,State -X
JobID JobName Elapsed MaxRSS ReqMem ReqCPUS CPUTime State
------------ ---------- ---------- ---------- ---------- -------- ---------- ----------
42_2 val_ja3 00:00:10 256M 1 00:00:10 CANCELLED+
42_3 val_ja3 00:00:10 256M 1 00:00:10 CANCELLED+
42 val_bp2a 00:00:16 24G 8 00:02:08 COMPLETED
43 val_bp2b 00:00:15 8G 4 00:01:00 COMPLETED
Exercise 3¶
Captured 2026-10-19T01:18:07Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct -j 42 -X --format=JobID,AllocCPUS,Elapsed,TotalCPU,State
JobID AllocCPUS Elapsed TotalCPU State
------------ ---------- ---------- ---------- ----------
42 2 00:00:04 00:00:00 COMPLETED
$ seff 42
Job ID: 42
Cluster: slurm-training-val
User/Group: ec2-user/ec2-user
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 2
CPU Utilized: 00:00:03
CPU Efficiency: 37.50% of 00:00:08 core-walltime
Job Wall-clock time: 00:00:04
Memory Utilized: 4.73 MB
Memory Efficiency: 0.02% of 30.40 GB (30.40 GB/node)
user/containers¶
Exercise 1¶
Captured 2026-10-19T01:18:09Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ module load apptainer/1.5.4 && apptainer --version
apptainer version 1.5.4
$ module load apptainer/1.5.4 && apptainer pull --force ubuntu.sif docker://public.ecr.aws/docker/library/ubuntu:latest 2>&1
INFO: Using cached SIF image
$ module load apptainer/1.5.4 && apptainer exec ubuntu.sif cat /etc/os-release
INFO: squashfuse not found, will not be able to mount SIF or other squashfs files
INFO: gocryptfs not found, will not be able to use gocryptfs
INFO: Converting SIF file to temporary sandbox...
PRETTY_NAME="Ubuntu 26.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="26.04"
VERSION="26.04.1 LTS (Resolute Raccoon)"
VERSION_CODENAME=resolute
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=resolute
LOGO=ubuntu-logo
INFO: Cleaning up image...
Exercise 2¶
Captured 2026-10-19T01:18:11Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
user/environment-modules¶
Exercise 1¶
Captured 2026-10-19T01:18:17Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ module avail testapp 2>&1
----------------------------- /shared/modulefiles ------------------------------
testapp/1.0 testapp/2.0 (D)
----------------------------- /shared/modulefiles ------------------------------
testapp/1.0 testapp/2.0 (D)
Where:
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".
$ module load testapp/1.0 && module list 2>&1
Currently Loaded Modules:
1) testapp/1.0
$ module unload testapp && module list 2>&1
No modules loaded
Exercise 2¶
Captured 2026-10-19T01:18:17Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ module purge && module load testapp/1.0 && module list 2>&1 && echo '---' && module swap testapp/1.0 testapp/2.0 && module list 2>&1
Currently Loaded Modules:
1) testapp/1.0
---
The following have been reloaded with a version change:
1) testapp/1.0 => testapp/2.0
Currently Loaded Modules:
1) testapp/2.0
Exercise 3¶
Captured 2026-10-19T01:18:18Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ module purge && module load testapp/2.0 && module save val_em3_collection 2>&1
Saved current collection of modules to: "val_em3_collection"
$ module purge && module list 2>&1
No modules loaded
$ module restore val_em3_collection && module list 2>&1
Restoring modules from user's val_em3_collection
Currently Loaded Modules:
1) testapp/2.0
user/getting-started¶
Exercise 1¶
Captured 2026-10-19T01:18:19Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 2¶
Captured 2026-10-19T01:18:24Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 3¶
Captured 2026-10-19T01:18:29Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 4¶
Captured 2026-10-19T01:18:35Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct -j 42 --format=JobID,JobName,Elapsed,State,ExitCode
JobID JobName Elapsed State ExitCode
------------ ---------- ---------- ---------- --------
42 val_gs4 00:00:01 COMPLETED 0:0
42.batch batch 00:00:01 COMPLETED 0:0
Exercise 5¶
Captured 2026-10-19T01:18:35Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up infinite 2 idle~ batch-dy-compute-[3-4]
batch* up infinite 2 idle batch-dy-compute-[1-2]
debug up infinite 2 idle~ debug-dy-small-[1-2]
gpu up infinite 1 idle~ gpu-dy-t4-1
highmem up infinite 1 idle~ highmem-dy-large-1
user/gpu-jobs¶
Exercise 1¶
Captured 2026-10-19T01:18:36Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --partition=gpu --gres=gpu:1 --cpus-per-task=4 --mem=8G --time=00:10:00 --job-name=val_gpu1 --output=gpu_%j.out --wrap='echo CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES; nvidia-smi'
Submitted batch job 42
$ cat gpu_42.out
CUDA_VISIBLE_DEVICES=0
Mon Oct 19 01:20:43 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 |
| N/A 24C P8 13W / 70W | 0MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Exercise 2¶
Captured 2026-10-19T01:20:47Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 3¶
Captured 2026-10-19T01:20:47Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --partition=gpu --gres=gpu:1 --cpus-per-task=4 --mem=8G --time=00:10:00 --job-name=val_gpu3 --output=gpu3_%j.out --wrap='echo CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES; nvidia-smi --query-gpu=index,name,memory.total --format=csv'
Submitted batch job 42
$ cat gpu3_42.out
CUDA_VISIBLE_DEVICES=0
index, name, memory.total [MiB]
0, Tesla T4, 15360 MiB
Exercise 4¶
Captured 2026-10-19T01:21:08Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct -j 42 --format=JobID,JobName,AllocTRES%60,Elapsed,State
JobID JobName AllocTRES Elapsed State
------------ ---------- ------------------------------------------------------------ ---------- ----------
42 val_gpu4 billing=4,cpu=4,gres/gpu=1,mem=8G,node=1 00:00:11 COMPLETED
42.batch batch cpu=4,gres/gpu=1,mem=8G,node=1 00:00:11 COMPLETED
user/interactive-jobs¶
Exercise 1¶
Captured 2026-10-19T01:21:09Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ srun --cpus-per-task=2 --mem=4G --time=00:05:00 hostname
batch-dy-compute-1
$ srun --cpus-per-task=2 --mem=4G --time=00:05:00 nproc
2
Exercise 2¶
Captured 2026-10-19T01:21:09Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ srun --cpus-per-task=4 --mem=8G --time=00:05:00 hostname
batch-dy-compute-1
$ srun --cpus-per-task=4 --mem=8G --time=00:05:00 free -m | head -2
total used free shared buff/cache available
Mem: 31720 472 30469 10 778 30846
Exercise 4¶
Captured 2026-10-19T01:21:10Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ srun --time=00:05:00 uname -a
Linux batch-dy-compute-1 6.12.103-127.188.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Aug 25 15:42:53 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
user/job-arrays¶
Exercise 1¶
Captured 2026-10-19T01:21:12Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 2¶
Captured 2026-10-19T01:21:17Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 3¶
Captured 2026-10-19T01:21:22Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --array=1-10%3 --time=00:05:00 --mem=256M --job-name=val_ja3 --output=/dev/null --wrap='sleep 30'
Submitted batch job 42
$ squeue -j 42 -h -t RUNNING -o '%i' | wc -l
3
Exercise 4¶
Captured 2026-10-19T01:21:37Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scancel 42_3
$ squeue -j 42 -r -o '%A_%a %T'
JOBID_ACCOUNT STATE
42_pcdefault RUNNING
153_pcdefault RUNNING
151_pcdefault RUNNING
150_pcdefault RUNNING
Exercise 5¶
Captured 2026-10-19T01:21:39Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
user/job-dependencies¶
Exercise 1¶
Captured 2026-10-19T01:21:45Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ squeue --me -o '%.8i %.20j %.2t %E'
JOBID NAME ST DEPENDENCY
159 val_jd1b PD afterok:158(unfulfilled)
158 val_jd1a PD (null)
Exercise 2¶
Captured 2026-10-19T01:21:51Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ squeue --me -o '%.8i %.20j %.2t %E'
JOBID NAME ST DEPENDENCY
162 val_jd2c PD afterok:160(unfulfilled),afterok:161(unfulfilled)
161 val_jd2b PD (null)
160 val_jd2a PD (null)
Exercise 3¶
Captured 2026-10-19T01:22:05Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ squeue --me -o '%.8i %.20j %.2t %R'
JOBID NAME ST NODELIST(REASON)
164 val_jd3_sg PD (Dependency)
163 val_jd3_sg R batch-dy-compute-1
Exercise 4¶
Captured 2026-10-19T01:22:15Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct -j 42,43,44 --format=JobID,JobName,State,ExitCode -X
JobID JobName State ExitCode
------------ ---------- ---------- --------
42 val_jd4f FAILED 1:0
43 val_jd4n PENDING 0:0
44 val_jd4h COMPLETED 0:0
$ squeue -j 43 -o '%.8i %.20j %.2t %.30r'
JOBID NAME ST REASON
43 val_jd4n PD DependencyNeverSatisfied
user/managing-jobs¶
Exercise 1¶
Captured 2026-10-19T01:22:16Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --time=01:00:00 --job-name=val_mg1 --output=/dev/null --wrap='sleep 3600'
Submitted batch job 42
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
42 batch val_mg1 ec2-user R 0:02 1 batch-dy-compute-1
$ scancel 42
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
Exercise 2¶
Captured 2026-10-19T01:22:20Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --hold --time=00:10:00 --job-name=val_mg2 --output=/dev/null --wrap='echo released'
Submitted batch job 42
$ squeue --me -o '%.8i %.20j %.2t %R'
JOBID NAME ST NODELIST(REASON)
42 val_mg2 PD (JobHeldUser)
$ scontrol release 42
$ squeue --me -o '%.8i %.20j %.2t %R'
JOBID NAME ST NODELIST(REASON)
Exercise 3¶
Captured 2026-10-19T01:22:24Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --hold --time=01:00:00 --job-name=val_mg3 --output=/dev/null --wrap='sleep 60'
Submitted batch job 42
$ scontrol update JobId=42 TimeLimit=02:00:00
$ scontrol show job 42 | grep TimeLimit
RunTime=00:00:00 TimeLimit=02:00:00 TimeMin=N/A
$ scontrol release 42
Exercise 4¶
Captured 2026-10-19T01:22:27Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --time=00:05:00 --job-name=val_mg4a --output=/dev/null --wrap='sleep 60'
Submitted batch job 42
$ sbatch --time=00:05:00 --job-name=val_mg4b --output=/dev/null --wrap='sleep 60'
Submitted batch job 43
$ sbatch --hold --time=00:05:00 --job-name=val_mg4p --output=/dev/null --wrap='sleep 60'
Submitted batch job 44
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
44 batch val_mg4p ec2-user PD 0:00 1 (JobHeldUser)
42 batch val_mg4a ec2-user R 0:01 1 batch-dy-compute-1
43 batch val_mg4b ec2-user R 0:01 1 batch-dy-compute-2
$ scancel --me --state=PENDING
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
42 batch val_mg4a ec2-user R 0:03 1 batch-dy-compute-1
43 batch val_mg4b ec2-user R 0:03 1 batch-dy-compute-2
$ scancel --me
user/monitoring-jobs¶
Exercise 1¶
Captured 2026-10-19T01:22:33Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ squeue --me -o '%.8i %.9P %.30j %.2t %.10M %.10l %R'
JOBID PARTITION NAME ST TIME TIME_LIMIT NODELIST(REASON)
171 batch val_mg4a CG 0:03 5:00 batch-dy-compute-1
172 batch val_mg4b CG 0:03 5:00 batch-dy-compute-2
174 batch val_mj1_format PD 0:00 10:00 (None)
Exercise 2¶
Captured 2026-10-19T01:22:35Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scontrol show job 42
JobId=42 JobName=val_mj2_inspect
UserId=ec2-user(1000) GroupId=ec2-user(1000) MCS_label=N/A
Priority=3500 Nice=0 Account=pcdefault QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:02 TimeLimit=00:10:00 TimeMin=N/A
SubmitTime=2026-10-19T01:22:33 EligibleTime=2026-10-19T01:22:33
AccrueTime=2026-10-19T01:22:33
StartTime=2026-10-19T01:22:33 EndTime=2026-10-19T01:32:33 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-10-19T01:22:33 Scheduler=Main
Partition=batch AllocNode:Sid=ip-172-31-84-168:110925
ReqNodeList=(null) ExcNodeList=(null)
NodeList=batch-dy-compute-2
BatchHost=batch-dy-compute-2
NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=2 ReqB:S:C:T=0:0:*:*
ReqTRES=cpu=2,mem=2G,node=1,billing=2
AllocTRES=cpu=2,mem=2G,node=1,billing=2
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=2 MinMemoryNode=2G MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null)
Command=(null)
SubmitLine=sbatch --time=00:10:00 --mem=2G --cpus-per-task=2 --wrap=sleep 300 --job-name=val_mj2_inspect
WorkDir=/home/ec2-user/slurm-val-MkHI0x
StdErr=
StdIn=/dev/null
StdOut=/home/ec2-user/slurm-val-MkHI0x/slurm-42.out
TresPerTask=cpu=2
Exercise 3¶
Captured 2026-10-19T01:22:45Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sacct -j 42 --format=JobID,JobName,ReqMem,MaxRSS,State
JobID JobName ReqMem MaxRSS State
------------ ---------- ---------- ---------- ----------
42 val_mj3 512M COMPLETED
42.batch batch 868K COMPLETED
Exercise 4¶
Captured 2026-10-19T01:22:45Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sinfo -p batch -t idle -o '%n %c %m'
HOSTNAMES CPUS MEMORY
batch-dy-compute-3 8 31129
batch-dy-compute-4 8 31129
batch-dy-compute-1 8 31129
batch-dy-compute-2 8 31129
Exercise 5¶
Captured 2026-10-19T01:22:48Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ squeue --me --start
JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON)
177 batch val_mj5 ec2-user PD N/A 1 (null) (JobHeldUser)
Exercise 6¶
Captured 2026-10-19T01:22:49Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --time=00:05:00 --wrap='sleep 120' --job-name=val_json_test
Submitted batch job 42
$ squeue --me --json | jq '.jobs[] | {job_id, job_state}'
{
"job_id": 42,
"job_state": [
"PENDING"
]
}
user/parallel-mpi-jobs¶
Exercise 1¶
Captured 2026-10-19T01:22:50Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 2¶
Captured 2026-10-19T01:22:55Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 3¶
Captured 2026-10-19T01:23:00Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 4¶
Captured 2026-10-19T01:23:05Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 5¶
Captured 2026-10-19T01:23:10Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --job-name=val_scale1 --cpus-per-task=1 --ntasks=1 --time=00:05:00 --output=scale1_%j.out --wrap='/shared/python/3.12/bin/python3 busy.py'
Submitted batch job 42
$ sbatch --job-name=val_scale2 --cpus-per-task=2 --ntasks=1 --time=00:05:00 --output=scale2_%j.out --wrap='/shared/python/3.12/bin/python3 busy.py'
Submitted batch job 43
$ sbatch --job-name=val_scale4 --cpus-per-task=4 --ntasks=1 --time=00:05:00 --output=scale4_%j.out --wrap='/shared/python/3.12/bin/python3 busy.py'
Submitted batch job 44
$ sacct -X --name=val_scale1,val_scale2,val_scale4 -o JobID,JobName,AllocCPUS,TotalCPU,Elapsed
JobID JobName AllocCPUS TotalCPU Elapsed
------------ ---------- ---------- ---------- ----------
86 val_scale1 1 00:00:00 00:00:06
87 val_scale2 2 00:00:00 00:00:05
88 val_scale4 4 00:00:00 00:00:02
42 val_scale1 1 00:00:00 00:00:04
43 val_scale2 2 00:00:00 00:00:03
44 val_scale4 4 00:00:00 00:00:02
user/recurring-jobs-scrontab¶
Exercise 1¶
Captured 2026-10-19T01:25:48Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 2¶
Captured 2026-10-19T01:25:48Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ scrontab -l
#SCRON --time=00:10:00
#SCRON --mem=1G
#SCRON -J val_scron_hourly
#SCRON --output=/home/ec2-user/logs/val_scron_hourly_%j.out
@hourly /home/ec2-user/scripts/val_scron_hourly.sh
$ squeue --me --name=val_scron_hourly || true
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
187 batch val_scro ec2-user PD 0:00 1 (BeginTime)
Exercise 3¶
Captured 2026-10-19T01:25:48Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
user/resource-requests¶
Exercise 1¶
Captured 2026-10-19T01:25:49Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 2¶
Captured 2026-10-19T01:25:54Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --exclusive --mem=0 --time=00:10:00 --job-name=val_rr2 --output=exclusive_%j.out --wrap='echo CPUs: $(nproc); free -g'
Submitted batch job 42
Exercise 3¶
Captured 2026-10-19T01:25:59Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sinfo -o '%n %f'
HOSTNAMES AVAIL_FEATURES
batch-dy-compute-4 dynamic,t3.2xlarge,compute
batch-dy-compute-1 dynamic,t3.2xlarge,compute
batch-dy-compute-2 dynamic,t3.2xlarge,compute
batch-dy-compute-3 dynamic,t3.2xlarge,compute
debug-dy-small-1 dynamic,t3.medium,small
debug-dy-small-2 dynamic,t3.medium,small
gpu-dy-t4-1 dynamic,g4dn.xlarge,t4,gpu
highmem-dy-large-1 dynamic,t3.2xlarge,large
Exercise 4¶
Captured 2026-10-19T01:25:59Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --time=00:05:00 --mem=16G --job-name=val_rr4 --output=/dev/null --wrap='sleep 10'
Submitted batch job 42
$ sacct -j 42 --format=JobID,JobName,ReqMem,MaxRSS,Elapsed,State
JobID JobName ReqMem MaxRSS Elapsed State
------------ ---------- ---------- ---------- ---------- ----------
42 val_rr4 16G 00:00:10 COMPLETED
42.batch batch 868K 00:00:10 COMPLETED
Exercise 5¶
Captured 2026-10-19T01:26:15Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --parsable --hold --time=00:05:00 --mem=16G --cpus-per-task=4 --job-name=val_rr5a --wrap='sleep 60'
191
$ sbatch --parsable --hold --time=00:05:00 --mem-per-cpu=4G --cpus-per-task=4 --job-name=val_rr5b --wrap='sleep 60'
192
$ scontrol show job 191 | grep -E '(MinMemory|TRES=)'
ReqTRES=cpu=4,mem=16G,node=1,billing=4
AllocTRES=(null)
MinCPUsNode=4 MinMemoryNode=16G MinTmpDiskNode=0
$ scontrol show job 192 | grep -E '(MinMemory|TRES=)'
ReqTRES=cpu=4,mem=16G,node=1,billing=4
AllocTRES=(null)
MinCPUsNode=4 MinMemoryCPU=4G MinTmpDiskNode=0
$ scancel 191 192
user/slurm-overview¶
Exercise 1¶
Captured 2026-10-19T01:26:18Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up infinite 1 idle~ batch-dy-compute-4
batch* up infinite 3 idle batch-dy-compute-[1-3]
debug up infinite 2 idle~ debug-dy-small-[1-2]
gpu up infinite 1 idle gpu-dy-t4-1
highmem up infinite 1 idle~ highmem-dy-large-1
$ sinfo --summarize
PARTITION AVAIL TIMELIMIT NODES(A/I/O/T) NODELIST
batch* up infinite 0/4/0/4 batch-dy-compute-[1-4]
debug up infinite 0/2/0/2 debug-dy-small-[1-2]
gpu up infinite 0/1/0/1 gpu-dy-t4-1
highmem up infinite 0/1/0/1 highmem-dy-large-1
Exercise 2¶
Captured 2026-10-19T01:26:18Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
user/submitting-jobs¶
Exercise 1¶
Captured 2026-10-19T01:26:19Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 2¶
Captured 2026-10-19T01:26:24Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --time=00:05:00 --mem=1G --job-name=val_sj2 --output=/dev/null --wrap='date; head -5 /proc/cpuinfo'
Submitted batch job 42
Exercise 3¶
Captured 2026-10-19T01:26:24Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --mail-type=BEGIN,END,FAIL --mail-user=you@example.com --time=00:05:00 --job-name=val_sj3 --wrap='sleep 30'
Submitted batch job 42
$ scontrol show job 42 | grep -i mail
SubmitLine=sbatch --mail-type=BEGIN,END,FAIL --mail-user=you@example.com --time=00:05:00 --job-name=val_sj3 --wrap=sleep 30
MailUser=you@example.com MailType=BEGIN,END,FAIL
Exercise 4¶
Captured 2026-10-19T01:26:26Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
Exercise 5¶
Captured 2026-10-19T01:26:31Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1
$ sbatch --chdir=/home/ec2-user/slurm-val-chdir --time=00:05:00 --job-name=val_sj5 --output=chdir_%j.out --wrap='pwd > chdir_test.txt'
Submitted batch job 42
$ cat /home/ec2-user/slurm-val-chdir/chdir_test.txt
/home/ec2-user/slurm-val-chdir
Exercise 6¶
Captured 2026-10-19T01:26:37Z · cluster slurm-training-val · Slurm 25.11.8 · ParallelCluster 3.16.1