Exercise Validation Details¶
Captured from slurm-training-val at 2026-04-15T01:40:12Z. See the
summary page for the full toolchain and
run-level context.
Each entry below shows the command(s) that the validation harness ran for that exercise and the real output that came back. A non-zero exit code is flagged with a comment line inside the code block — some exercises intentionally trigger a failure (e.g., resource over-request demonstrations).
admin/accounts-fairshare¶
Exercise 1¶
Captured 2026-04-15T01:40:23Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sudo /opt/slurm/bin/sacctmgr -i add account val_research Description='Research Division'
Adding Account(s)
val_research
Settings
Description = research division
Organization = Parent/Account Name
Associations =
C = slurm-training-val A = val_research
Settings
$ sudo /opt/slurm/bin/sacctmgr -i add account val_structural_biology parent=val_research Description='Structural Biology Dept'
Adding Account(s)
val_structural_biology
Settings
Description = structural biology dept
Organization = Parent/Account Name
Associations =
C = slurm-training-val A = val_structural_biology
Settings
Parent = val_research
$ sudo /opt/slurm/bin/sacctmgr -i add account val_cryo_lab parent=val_structural_biology Description='Cryo-EM Lab'
Adding Account(s)
val_cryo_lab
Settings
Description = cryo-em lab
Organization = Parent/Account Name
Associations =
C = slurm-training-val A = val_cryo_lab
Settings
Parent = val_structural_biology
$ sudo /opt/slurm/bin/sacctmgr -i add user val_pi_chen account=val_cryo_lab fairshare=20 qos=normal defaultqos=normal
Adding User(s)
val_pi_chen
Settings
Associations =
C = slurm-training-val A = val_cryo_lab U = val_pi_chen
Non Default Settings
Fairshare = 20
QOS = normal
DefQOS = normal
$ sudo /opt/slurm/bin/sacctmgr -i add user val_postdoc_kim account=val_cryo_lab fairshare=10 qos=normal defaultqos=normal
Adding User(s)
val_postdoc_kim
Settings
Associations =
C = slurm-training-val A = val_cryo_lab U = val_postdoc_kim
Non Default Settings
Fairshare = 10
QOS = normal
DefQOS = normal
$ sacctmgr show association where account=val_cryo_lab format=Account,User,Fairshare tree withd
Account User Share
-------------------- ---------- ---------
val_cryo_lab 1
val_cryo_lab val_pi_ch+ 20
val_cryo_lab val_postd+ 10
$ sacctmgr show user val_pi_chen,val_postdoc_kim format=User%-20,DefaultAccount%-20
User Def Acct
-------------------- --------------------
val_pi_chen val_cryo_lab
val_postdoc_kim val_cryo_lab
Exercise 2¶
Captured 2026-04-15T01:40:24Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sshare -l | head -20
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 6606 1.000000 cpu=0,mem=0,energy=0,node=0,b+
pcdefault 1 0.006579 6343 0.960190 0.960190 0.006852 cpu=0,mem=0,energy=0,node=0,b+
pcdefault ec2-user 1 0.500000 6343 0.960190 1.000000 0.250000 0.500000 cpu=0,mem=0,energy=0,node=0,b+
research 50 0.328947 263 0.039810 0.039810 8.262879 cpu=0,mem=0,energy=0,node=0,b+
training 100 0.657895 0 0.000000 0.000000 inf cpu=0,mem=0,energy=0,node=0,b+
Exercise 3¶
Captured 2026-04-15T01:40:27Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sudo /opt/slurm/bin/sacctmgr -i modify account val_structural_biology set fairshare=40
Modified account associations...
C = slurm-training-val A = val_structural_biology of val_research
$ sudo /opt/slurm/bin/sacctmgr -i modify account val_chemistry set fairshare=60
Modified account associations...
C = slurm-training-val A = val_chemistry of val_research
$ sacctmgr show account val_structural_biology,val_chemistry format=Account%-25,Description%-25,Organization%-25
Account Descr Org
------------------------- ------------------------- -------------------------
val_chemistry val_chemistry val_research
val_structural_biology val_structural_biology val_research
Exercise 4¶
Captured 2026-04-15T01:40:27Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sreport cluster AccountUtilizationByUser start=now-7days --tres=cpu | head -15
--------------------------------------------------------------------------------
Cluster/Account/User Utilization 2026-04-08T01:00:00 - 2026-04-15T00:59:59 (604800 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
Cluster Account Login Proper Name TRES Name Used
--------- --------------- --------- --------------- -------------- --------
slurm-tr+ root cpu 286
slurm-tr+ pcdefault cpu 281
slurm-tr+ pcdefault ec2-user AWS ParallelCl+ cpu 281
slurm-tr+ research cpu 5
slurm-tr+ research testuser1 cpu 5
$ sreport user TopUsage start=now-7days --tres=cpu TopCount=10 | head -15
--------------------------------------------------------------------------------
Top 10 Users 2026-04-08T01:00:00 - 2026-04-15T00:59:59 (604800 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
Cluster Login Proper Name Account TRES Name Used
--------- --------- --------------- --------------- -------------- --------
slurm-tr+ ec2-user AWS ParallelCl+ pcdefault cpu 281
slurm-tr+ testuser1 research cpu 5
admin/configuration¶
Exercise 1¶
Captured 2026-04-15T01:40:29Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 2¶
Captured 2026-04-15T01:40:29Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sudo /opt/slurm/bin/scontrol setdebug debug2
$ scontrol show config | grep SlurmctldDebug
SlurmctldDebug = debug2
$ scontrol show config | grep SlurmctldDebug
SlurmctldDebug = info
Exercise 3¶
Captured 2026-04-15T01:40:30Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scontrol show partition val_extra
PartitionName=val_extra
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=NO QoS=N/A
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
Nodes=val_node1
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=1 TotalNodes=1 SelectTypeParameters=NONE
JobDefaults=(null)
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
TRES=cpu=1,mem=128M,node=1,billing=1
ResumeTimeout=GLOBAL SuspendTimeout=GLOBAL SuspendTime=GLOBAL PowerDownOnIdle=NO
$ scontrol show node val_node1 || true
Node val_node1 not found
$ sudo grep '^NodeName=val_node1' /opt/slurm/etc/slurm.conf
NodeName=val_node1 CPUs=1 RealMemory=128 State=FUTURE
Exercise 4¶
Captured 2026-04-15T01:40:31Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scontrol show partition val_extra2
PartitionName=val_extra2
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=NO QoS=N/A
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
Nodes=val_node2
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=2 TotalNodes=1 SelectTypeParameters=NONE
JobDefaults=(null)
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
TRES=cpu=2,mem=256M,node=1,billing=2
ResumeTimeout=GLOBAL SuspendTimeout=GLOBAL SuspendTime=GLOBAL PowerDownOnIdle=NO
admin/high-availability¶
Exercise 1¶
Captured 2026-04-15T01:40:32Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scontrol show config | grep SlurmctldHost
SlurmctldHost[0] = ip-172-31-94-160(172.31.94.160)
$ scontrol show config | grep StateSaveLocation
StateSaveLocation = /var/spool/slurm.state
Exercise 2¶
Captured 2026-04-15T01:40:32Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scontrol ping
Slurmctld(primary) at ip-172-31-94-160 is UP
$ sinfo --summarize
PARTITION AVAIL TIMELIMIT NODES(A/I/O/T) NODELIST
batch* up infinite 0/4/0/4 batch-dy-compute-[1-4]
debug up infinite 0/2/0/2 debug-dy-small-[1-2]
gpu up infinite 0/1/0/1 gpu-dy-t4-1
highmem up infinite 0/1/0/1 highmem-dy-large-1
admin/installation¶
Exercise 1¶
Captured 2026-04-15T01:40:33Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ munge -n | unmunge | head -5
STATUS: Success (0)
ENCODE_HOST: ip-172-31-94-160.us-east-2.compute.internal (172.31.94.160)
ENCODE_TIME: 2026-04-15 01:40:33 +0000 (1776217233)
DECODE_TIME: 2026-04-15 01:40:33 +0000 (1776217233)
TTL: 300
Exercise 2¶
Captured 2026-04-15T01:40:33Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ systemctl is-active slurmdbd
active
$ sacctmgr -nP show cluster format=Cluster,ControlHost,ControlPort,RPC
slurm-training-val|172.31.94.160|6820|11264
Exercise 3¶
Captured 2026-04-15T01:40:33Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sacctmgr show cluster format=Cluster,ControlHost,ControlPort,RPC
Cluster ControlHost ControlPort RPC
---------- --------------- ------------ -----
slurm-tra+ 172.31.94.160 6820 11264
$ sacctmgr show account format=Account,Description,Organization
Account Descr Org
---------- -------------------- --------------------
pcdefault parallelcluster def+ none
research research bioteam
root default root account root
training training bioteam
$ sacctmgr show association format=Cluster,Account,User,Fairshare | head -15
Cluster Account User Share
---------- ---------- ---------- ---------
slurm-tra+ root 1
slurm-tra+ root root 1
slurm-tra+ pcdefault 1
slurm-tra+ pcdefault ec2-user 1
slurm-tra+ pcdefault slurm 1
slurm-tra+ research 50
slurm-tra+ research testuser1 1
slurm-tra+ training 100
admin/maintenance-operations¶
Exercise 1¶
Captured 2026-04-15T01:40:34Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sudo /opt/slurm/bin/scontrol update NodeName=debug-dy-small-1 State=DRAIN Reason='val: maintenance test'
$ scontrol show node debug-dy-small-1 | grep -E '(State|Reason)'
State=IDLE+CLOUD+DRAIN+POWERED_DOWN+NOT_RESPONDING ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
Reason=val: maintenance test [root@2026-04-15T01:40:34]
$ sudo /opt/slurm/bin/scontrol update NodeName=debug-dy-small-1 State=RESUME
$ scontrol show node debug-dy-small-1 | grep State=
State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
Exercise 2¶
Captured 2026-04-15T01:40:34Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sinfo -R
REASON USER TIMESTAMP NODELIST
$ sinfo -o '%T %D' --noheader | sort | uniq -c | sort -rn
1 idle~ 5
1 idle% 1
1 idle 2
$ sinfo -o '%T %C' --noheader | head -10
idle~ 0/28/0/28
idle 0/16/0/16
idle% 0/4/0/4
admin/monitoring-accounting¶
Exercise 1¶
Captured 2026-04-15T01:40:35Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sreport cluster utilization start=now-7days
--------------------------------------------------------------------------------
Cluster Utilization 2026-04-08T01:00:00 - 2026-04-15T00:59:59
Usage reported in CPU Minutes
--------------------------------------------------------------------------------
Cluster Allocate Down PLND Dow Idle Planned Reported
--------- -------- -------- -------- -------- -------- --------
slurm-tr+ 286 410 109206 7404 505 117811
Exercise 2¶
Captured 2026-04-15T01:40:35Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sreport user TopUsage start=now-7days --tres=cpu TopCount=5
--------------------------------------------------------------------------------
Top 5 Users 2026-04-08T01:00:00 - 2026-04-15T00:59:59 (604800 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
Cluster Login Proper Name Account TRES Name Used
--------- --------- --------------- --------------- -------------- --------
slurm-tr+ ec2-user AWS ParallelCl+ pcdefault cpu 281
slurm-tr+ testuser1 research cpu 5
Exercise 3¶
Captured 2026-04-15T01:40:35Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sdiag
*******************************************************
sdiag output at Wed Apr 15 01:40:35 2026 (1776217235)
Data since Wed Apr 15 01:40:31 2026 (1776217231)
*******************************************************
Server thread count: 1
RPC queue enabled: 0
Agent queue size: 0
Agent count: 0
Agent thread count: 0
DBD Agent queue size: 0
Jobs submitted: 0
Jobs started: 0
Jobs completed: 0
Jobs canceled: 0
Jobs failed: 0
Job states ts: Wed Apr 15 01:40:31 2026 (1776217231)
Jobs pending: 0
Jobs running: 0
Main schedule statistics (microseconds):
Last cycle: 13
Max cycle: 13
Total cycles: 1
Mean cycle: 13
Mean depth cycle: 0
Last queue length: 0
Main scheduler exit:
End of job queue: 1
Hit default_queue_depth: 0
Hit sched_max_job_start: 0
Blocked on licenses: 0
Hit max_rpc_cnt: 0
Timeout (max_sched_time): 0
Backfilling stats
Total backfilled jobs (since last slurm start): 0
Total backfilled jobs (since last stats cycle start): 0
Total backfilled heterogeneous job components: 0
Total cycles: 0
Last cycle when: Wed Apr 15 01:32:22 2026 (1776216742)
Last cycle: 0
Max cycle: 0
Last depth cycle: 0
Last depth cycle (try sched): 0
Last queue length: 0
Last table size: 0
Backfill exit
End of job queue: 0
Hit bf_max_job_start: 0
Hit bf_max_job_test: 0
System state changed: 0
Hit table size limit (bf_node_space_size): 0
Timeout (bf_max_time): 0
Latency for 1000 calls to gettimeofday(): 29 microseconds
Remote Procedure Call statistics by message type
REQUEST_PARTITION_INFO ( 2009) count:6 ave_time:61 total_time:367
REQUEST_NODE_INFO ( 2007) count:6 ave_time:151 total_time:907
REQUEST_UPDATE_NODE ( 3002) count:4 ave_time:134 total_time:539
REQUEST_PING ( 1008) count:3 ave_time:42 total_time:127
MESSAGE_NODE_REGISTRATION_STATUS ( 1002) count:2 ave_time:114 total_time:228
REQUEST_BUILD_INFO ( 2001) count:2 ave_time:154 total_time:309
Remote Procedure Call statistics by user
ec2-user ( 1000) count:17 ave_time:100 total_time:1710
root ( 0) count:6 ave_time:127 total_time:767
Pending RPC statistics
No pending RPCs
Exercise 4¶
Captured 2026-04-15T01:40:35Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sacct --starttime=now-7days --state=OUT_OF_MEMORY --format=JobID,User,JobName,ReqMem,MaxRSS,Elapsed,State,ExitCode
JobID User JobName ReqMem MaxRSS Elapsed State ExitCode
------------ --------- ---------- ---------- ---------- ---------- ---------- --------
$ sacct --starttime=now-7days --state=COMPLETED --format=JobID%-12,User%-10,JobName%-20,ReqMem,MaxRSS,Elapsed | head -10
JobID User JobName ReqMem MaxRSS Elapsed
------------ ---------- -------------------- ---------- ---------- ----------
admin/partitions-qos¶
Exercise 1¶
Captured 2026-04-15T01:40:37Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scontrol show partition val_long
PartitionName=val_long
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=NO QoS=N/A
DefaultTime=04:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=14-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
Nodes=debug-dy-small-[1-2]
PriorityJobFactor=1 PriorityTier=50 RootOnly=NO ReqResv=NO OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=4 TotalNodes=2 SelectTypeParameters=NONE
JobDefaults=(null)
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
TRES=cpu=4,mem=7782M,node=2,billing=4
ResumeTimeout=GLOBAL SuspendTimeout=GLOBAL SuspendTime=GLOBAL PowerDownOnIdle=NO
$ sinfo -p val_long
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
val_long up 14-00:00:0 2 idle~ debug-dy-small-[1-2]
Exercise 2¶
Captured 2026-04-15T01:40:38Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sudo /opt/slurm/bin/sacctmgr -i add qos val_pq_restricted set MaxTRESPerUser=cpu=32 MaxJobsPerUser=10 Priority=0
Adding QOS(s)
val_pq_restricted
Settings
Description = val_pq_restricted
MaxJobsPerUser = 10
MaxTRESPerUser = cpu=32
Priority = 0
$ sacctmgr show qos val_pq_restricted format=Name%-20,MaxTRESPerUser%-30,MaxJobsPerUser,Priority
Name MaxTRESPU MaxJobsPU Priority
-------------------- ------------------------------ --------- ----------
val_pq_restricted cpu=32 10 0
Exercise 3¶
Captured 2026-04-15T01:40:39Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sudo /opt/slurm/bin/sacctmgr -i modify account training set qos=normal,val_pq_restricted defaultqos=normal
Modified account associations...
C = slurm-training-val A = training of root
$ sacctmgr show association where account=training format=Account,User,QOS,DefaultQOS
Account User QOS Def QOS
---------- ---------- -------------------- ---------
training normal,val_pq_restr+ normal
Exercise 4¶
Captured 2026-04-15T01:40:55Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ squeue -u testuser1 -o '%.8i %.2t %.30R'
JOBID ST NODELIST(REASON)
530 PD (QOSMaxCpuPerUserLimit)
529 R batch-dy-compute-2
528 R batch-dy-compute-1
admin/policies-priority¶
Exercise 1¶
Captured 2026-04-15T01:40:58Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sprio -l
JOBID PARTITION USER ACCOUNT PRIORITY SITE AGE ASSOC FAIRSHARE JOBSIZE PARTITION QOSNAME QOS NICE TRES
$ sprio -n -l
JOBID PARTITION USER ACCOUNT PRIORITY AGE ASSOC FAIRSHARE JOBSIZE PARTITION QOSNAME QOS TRES
$ sshare -a -l | head -10
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 6624 1.000000 cpu=0,mem=0,energy=0,node=0,b+
root root 1 0.006579 0 0.000000 0.000000 1.000000 inf cpu=0,mem=0,energy=0,node=0,b+
pcdefault 1 0.006579 6343 0.957581 0.957581 0.006870 cpu=0,mem=0,energy=0,node=0,b+
pcdefault ec2-user 1 0.500000 6343 0.957581 1.000000 0.250000 0.500000 cpu=0,mem=0,energy=0,node=0,b+
pcdefault slurm 1 0.500000 0 0.000000 0.000000 0.500000 inf cpu=0,mem=0,energy=0,node=0,b+
research 50 0.328947 281 0.042419 0.042419 7.754679 cpu=0,mem=0,energy=0,node=0,b+
research testuser1 1 1.000000 281 0.042419 1.000000 0.750000 1.000000 cpu=0,mem=0,energy=0,node=0,b+
training 100 0.657895 0 0.000000 0.000000 inf cpu=0,mem=0,energy=0,node=0,b+
Exercise 2¶
Captured 2026-04-15T01:40:58Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sudo /opt/slurm/bin/scontrol create reservation ReservationName=val_pp_resv StartTime=now+3600 Duration=00:30:00 Nodes=debug-dy-small-1 Users=ec2-user Flags=MAINT,IGNORE_JOBS
Reservation created: val_pp_resv
$ scontrol show reservation val_pp_resv
ReservationName=val_pp_resv StartTime=2026-04-15T02:40:58 EndTime=2026-04-15T03:10:58 Duration=00:30:00
Nodes=debug-dy-small-1 NodeCnt=1 CoreCnt=2 Features=(null) PartitionName=(null) Flags=MAINT,IGNORE_JOBS,SPEC_NODES
TRES=cpu=2
AllowedPartitions=(null) QOS=(null)
Users=ec2-user Groups=(null) Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null)
MaxStartDelay=(null)
$ sudo /opt/slurm/bin/scontrol delete ReservationName=val_pp_resv
Exercise 3¶
Captured 2026-04-15T01:40:59Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sdiag | head -40
*******************************************************
sdiag output at Wed Apr 15 01:40:59 2026 (1776217259)
Data since Wed Apr 15 01:40:55 2026 (1776217255)
*******************************************************
Server thread count: 1
RPC queue enabled: 0
Agent queue size: 0
Agent count: 1
Agent thread count: 4
DBD Agent queue size: 0
Jobs submitted: 1
Jobs started: 0
Jobs completed: 0
Jobs canceled: 1
Jobs failed: 0
Job states ts: Wed Apr 15 01:40:55 2026 (1776217255)
Jobs pending: 0
Jobs running: 0
Main schedule statistics (microseconds):
Last cycle: 10
Max cycle: 29
Total cycles: 3
Mean cycle: 16
Mean depth cycle: 0
Last queue length: 0
Main scheduler exit:
End of job queue: 3
Hit default_queue_depth: 0
Hit sched_max_job_start: 0
Blocked on licenses: 0
Hit max_rpc_cnt: 0
Timeout (max_sched_time): 0
Backfilling stats
Total backfilled jobs (since last slurm start): 0
Total backfilled jobs (since last stats cycle start): 0
admin/resource-management¶
Exercise 1¶
Captured 2026-04-15T01:41:11Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ echo 'NodeName=batch-dy-compute-1 CPUs=8 RealMemory=31129 State=CLOUD Feature=dynamic,t3.2xlarge,compute Weight=1000 Gres=val_fpga:2'
NodeName=batch-dy-compute-1 CPUs=8 RealMemory=31129 State=CLOUD Feature=dynamic,t3.2xlarge,compute Weight=1000 Gres=val_fpga:2
$ echo 'NodeName=batch-dy-compute-1 Name=val_fpga Count=2'
NodeName=batch-dy-compute-1 Name=val_fpga Count=2
$ ssh batch-dy-compute-1 'sudo systemctl restart slurmd'
Warning: Permanently added 'batch-dy-compute-1' (ED25519) to the list of known hosts.
$ grep val_fpga /opt/slurm/etc/pcluster/slurm_parallelcluster_batch_partition.conf /opt/slurm/etc/pcluster/slurm_parallelcluster_batch_gres.conf
/opt/slurm/etc/pcluster/slurm_parallelcluster_batch_gres.conf:NodeName=batch-dy-compute-1 Name=val_fpga Count=2
$ scontrol show node batch-dy-compute-1 | grep -E '(NodeName|Gres|CfgTRES)'
NodeName=batch-dy-compute-1 Arch=x86_64 CoresPerSocket=1
Gres=(null)
CfgTRES=cpu=8,mem=31129M,billing=8
Exercise 2¶
Captured 2026-04-15T01:41:18Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scontrol show node gpu-dy-t4-1 | grep -E '(NodeName|Gres|CfgTRES|AllocTRES)'
NodeName=gpu-dy-t4-1 CoresPerSocket=1
Gres=gpu:t4:1
CfgTRES=cpu=4,mem=15564M,billing=4
AllocTRES=
Exercise 3¶
Captured 2026-04-15T01:41:20Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scontrol show licenses
LicenseName=val_glide
Total=5 Used=0 Free=5 Reserved=0 Remote=no
$ scontrol show licenses
LicenseName=val_glide
Total=5 Used=0 Free=5 Reserved=0 Remote=no
Exercise 4¶
Captured 2026-04-15T01:41:23Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ cat /opt/slurm/etc/cgroup.conf 2>/dev/null || echo '(cgroup.conf not at /opt/slurm/etc/)'
###
# Slurm cgroup support configuration file
###
ConstrainCores=yes
#
# WARNING!!! The slurm_parallelcluster_cgroup.conf file included below can be updated by the pcluster process.
# Please do not edit it.
include slurm_parallelcluster_cgroup.conf
$ sbatch --mem=100M --time=00:05:00 --output=/dev/null --job-name=val_oom --wrap='/shared/python/3.12/bin/python3 -c "x = bytearray(500 * 1024 * 1024); import time; time.sleep(60)"'
Submitted batch job 42
$ sacct -j 42 --format=JobID,JobName%-12,State%-20,ExitCode,MaxRSS
JobID JobName State ExitCode MaxRSS
------------ ------------ -------------------- -------- ----------
42 val_oom OUT_OF_MEMORY 0:125
42.batch batch OUT_OF_MEMORY 0:125 102204K
Exercise 5¶
Captured 2026-04-15T01:46:26Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scontrol show nodes | grep -E '(NodeName|AvailableFeatures)' | head -20
NodeName=batch-dy-compute-1 Arch=x86_64 CoresPerSocket=1
AvailableFeatures=dynamic,t3.2xlarge,compute
NodeName=batch-dy-compute-2 Arch=x86_64 CoresPerSocket=1
AvailableFeatures=dynamic,t3.2xlarge,compute
NodeName=batch-dy-compute-3 CoresPerSocket=1
AvailableFeatures=dynamic,t3.2xlarge,compute
NodeName=batch-dy-compute-4 CoresPerSocket=1
AvailableFeatures=dynamic,t3.2xlarge,compute
NodeName=debug-dy-small-1 CoresPerSocket=1
AvailableFeatures=dynamic,t3.medium,small
NodeName=debug-dy-small-2 CoresPerSocket=1
AvailableFeatures=dynamic,t3.medium,small
NodeName=gpu-dy-t4-1 CoresPerSocket=1
AvailableFeatures=dynamic,g4dn.xlarge,t4,gpu
NodeName=highmem-dy-large-1 CoresPerSocket=1
AvailableFeatures=dynamic,t3.2xlarge,large
admin/slurm-architecture¶
Exercise 1¶
Captured 2026-04-15T01:46:27Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ systemctl is-active slurmctld slurmdbd
active
active
$ ps -eo comm,pid | grep -E 'slurm(ctld|dbd|d|restd)' | head -10
slurmdbd 180649
slurmctld 180673
Exercise 2¶
Captured 2026-04-15T01:46:27Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ slurmctld -V
slurm 25.11.4
$ scontrol show config | grep SLURM_VERSION
SLURM_VERSION = 25.11.4
$ scontrol ping
Slurmctld(primary) at ip-172-31-94-160 is UP
admin/troubleshooting¶
Exercise 1¶
Captured 2026-04-15T01:46:30Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ squeue -j 42 -o '%.8i %.9P %.20j %.2t %.30R'
JOBID PARTITION NAME ST NODELIST(REASON)
42 batch val_ts1 PD (JobHeldUser)
$ scontrol show job 42 | grep -E '(JobState|Reason)'
JobState=PENDING Reason=JobHeldUser Dependency=(null)
Exercise 2¶
Captured 2026-04-15T01:46:30Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sudo /opt/slurm/bin/scontrol update NodeName=debug-dy-small-2 State=DRAIN Reason='val: ts2 drain test'
$ sinfo -n debug-dy-small-2
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up infinite 0 n/a
debug up infinite 1 drain~ debug-dy-small-2
gpu up infinite 0 n/a
highmem up infinite 0 n/a
$ scontrol show node debug-dy-small-2 | grep -E '(State|Reason)'
State=IDLE+CLOUD+DRAIN+POWERED_DOWN+NOT_RESPONDING ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
Reason=val: ts2 drain test [root@2026-04-15T01:46:30]
Exercise 3¶
Captured 2026-04-15T01:46:31Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sudo /opt/slurm/bin/scontrol update NodeName=debug-dy-small-2 State=RESUME
$ scontrol show node debug-dy-small-2 | grep State=
State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
Exercise 4¶
Captured 2026-04-15T01:46:34Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sacct --starttime=now-7days --state=OUT_OF_MEMORY --format=JobID%-12,User%-10,JobName%-20,ReqMem,MaxRSS,Elapsed,ExitCode,NodeList
JobID User JobName ReqMem MaxRSS Elapsed ExitCode NodeList
------------ ---------- -------------------- ---------- ---------- ---------- -------- ---------------
$ sacct --starttime=now-7days --format=JobID,User,State,ExitCode | grep '0:9' | head -10
$ sacct --starttime=now-7days --format=JobID,User,State,ExitCode | grep '0:9' | head -10 || echo '(no SIGKILL exits in window)'
Exercise 5¶
Captured 2026-04-15T01:46:36Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sinfo -N -l | head -20
Wed Apr 15 01:46:36 2026
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
batch-dy-compute-1 1 batch* idle 8 8:1:1 31129 0 1000 dynamic, none
batch-dy-compute-2 1 batch* idle 8 8:1:1 31129 0 1000 dynamic, none
batch-dy-compute-3 1 batch* idle~ 8 8:1:1 31129 0 1000 dynamic, none
batch-dy-compute-4 1 batch* idle~ 8 8:1:1 31129 0 1000 dynamic, none
debug-dy-small-1 1 debug idle~ 2 2:1:1 3891 0 1000 dynamic, none
debug-dy-small-2 1 debug idle~ 2 2:1:1 3891 0 1000 dynamic, none
gpu-dy-t4-1 1 gpu idle~ 4 4:1:1 15564 0 1000 dynamic, none
highmem-dy-large-1 1 highmem idle~ 8 8:1:1 31129 0 1000 dynamic, none
$ sinfo -R --noheader || echo '(no nodes with REASON set)'
deployment/aws-troubleshooting¶
Exercise 1¶
Captured 2026-04-15T01:46:37Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sinfo -R
REASON USER TIMESTAMP NODELIST
$ scontrol show node batch-dy-compute-1 | grep -E '(Reason|State)='
State=IDLE+CLOUD ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
Exercise 2¶
Captured 2026-04-15T01:46:37Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up infinite 2 idle~ batch-dy-compute-[3-4]
batch* up infinite 2 idle batch-dy-compute-[1-2]
debug up infinite 2 idle~ debug-dy-small-[1-2]
gpu up infinite 1 idle~ gpu-dy-t4-1
highmem up infinite 1 idle~ highmem-dy-large-1
$ sudo grep 'Node bootstrap error' /var/log/parallelcluster/clustermgtd 2>/dev/null | tail -5 || echo '(no bootstrap errors found)'
2026-04-14 22:10:56,42 - [slurm_plugin.slurm_resources:is_bootstrap_failure] - WARNING - Node bootstrap error: Node gpu-dy-t4-1(172.31.89.235) failed to register to the Slurm management daemon, node state: IDLE+CLOUD+DRAIN+INVALID_REG
2026-04-14 23:25:44,43 - [slurm_plugin.slurm_resources:is_bootstrap_failure] - WARNING - Node bootstrap error: Node gpu-dy-t4-1(172.31.87.111) failed to register to the Slurm management daemon, node state: IDLE+CLOUD+DRAIN+INVALID_REG
2026-04-15 00:12:44,44 - [slurm_plugin.slurm_resources:is_bootstrap_failure] - WARNING - Node bootstrap error: Node gpu-dy-t4-1(172.31.80.161) failed to register to the Slurm management daemon, node state: IDLE+CLOUD+DRAIN+INVALID_REG
$ sudo tail -3 /var/log/parallelcluster/clustermgtd
2026-04-15 01:45:43,45 - [slurm_plugin.clustermgtd:_maintain_nodes] - INFO - Performing node maintenance actions
2026-04-15 01:45:43,46 - [slurm_plugin.clustermgtd:_maintain_nodes] - INFO - Following nodes are currently in replacement: (x0) []
2026-04-15 01:45:43,46 - [slurm_plugin.clustermgtd:_terminate_orphaned_instances] - INFO - Checking for orphaned instance
user/best-practices¶
Exercise 1¶
Captured 2026-04-15T01:47:13Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sacct -j 42 --format=JobID,JobName,Elapsed,MaxRSS,MaxVMSize,CPUTime,ReqMem,ReqCPUS,State
JobID JobName Elapsed MaxRSS MaxVMSize CPUTime ReqMem ReqCPUS State
------------ ---------- ---------- ---------- ---------- ---------- ---------- -------- ----------
42 val_bp1 00:00:31 00:02:04 8G 4 COMPLETED
42.batch batch 00:00:31 1200K 0 00:02:04 4 COMPLETED
Exercise 2¶
Captured 2026-04-15T01:47:33Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sacct -j 42,43 --format=JobID,JobName,Elapsed,MaxRSS,ReqMem,ReqCPUS,CPUTime,State -X
JobID JobName Elapsed MaxRSS ReqMem ReqCPUS CPUTime State
------------ ---------- ---------- ---------- ---------- -------- ---------- ----------
42_2 val_ja5 00:00:00 1G 1 00:00:00 COMPLETED
42_3 val_ja5 00:00:00 1G 1 00:00:00 COMPLETED
42 val_bp2a 00:00:16 24G 8 00:02:08 COMPLETED
43 val_bp2b 00:00:16 8G 4 00:01:04 COMPLETED
user/containers¶
Exercise 1¶
Captured 2026-04-15T01:47:34Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ module load apptainer/1.3.6 && apptainer --version
apptainer version 1.3.6
$ module load apptainer/1.3.6 && apptainer pull --force ubuntu.sif docker://public.ecr.aws/docker/library/ubuntu:latest 2>&1
INFO: Using cached SIF image
$ module load apptainer/1.3.6 && apptainer exec ubuntu.sif cat /etc/os-release
INFO: squashfuse not found, will not be able to mount SIF or other squashfs files
INFO: gocryptfs not found, will not be able to use gocryptfs
INFO: Converting SIF file to temporary sandbox...
PRETTY_NAME="Ubuntu 24.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.4 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
INFO: Cleaning up image...
Exercise 2¶
Captured 2026-04-15T01:47:36Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
user/environment-modules¶
Exercise 1¶
Captured 2026-04-15T01:47:42Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ module avail testapp 2>&1
----------------------------- /shared/modulefiles ------------------------------
testapp/1.0 testapp/2.0 (D)
----------------------------- /shared/modulefiles ------------------------------
testapp/1.0 testapp/2.0 (D)
Where:
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".
$ module load testapp/1.0 && module list 2>&1
Currently Loaded Modules:
1) testapp/1.0
$ module unload testapp && module list 2>&1
No modules loaded
Exercise 2¶
Captured 2026-04-15T01:47:42Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ module purge && module load testapp/1.0 && module list 2>&1 && echo '---' && module swap testapp/1.0 testapp/2.0 && module list 2>&1
Currently Loaded Modules:
1) testapp/1.0
---
The following have been reloaded with a version change:
1) testapp/1.0 => testapp/2.0
Currently Loaded Modules:
1) testapp/2.0
Exercise 3¶
Captured 2026-04-15T01:47:42Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ module purge && module load testapp/2.0 && module save val_em3_collection 2>&1
Saved current collection of modules to: "val_em3_collection"
$ module purge && module list 2>&1
No modules loaded
$ module restore val_em3_collection && module list 2>&1
Restoring modules from user's val_em3_collection
Currently Loaded Modules:
1) testapp/2.0
user/getting-started¶
Exercise 1¶
Captured 2026-04-15T01:47:44Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 2¶
Captured 2026-04-15T01:47:49Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 3¶
Captured 2026-04-15T01:47:54Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 4¶
Captured 2026-04-15T01:47:59Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sacct -j 42 --format=JobID,JobName,Elapsed,State,ExitCode
JobID JobName Elapsed State ExitCode
------------ ---------- ---------- ---------- --------
42 val_gs4 00:00:01 COMPLETED 0:0
42.batch batch 00:00:01 COMPLETED 0:0
Exercise 5¶
Captured 2026-04-15T01:47:59Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up infinite 2 idle~ batch-dy-compute-[3-4]
batch* up infinite 2 idle batch-dy-compute-[1-2]
debug up infinite 2 idle~ debug-dy-small-[1-2]
gpu up infinite 1 idle~ gpu-dy-t4-1
highmem up infinite 1 idle~ highmem-dy-large-1
user/gpu-jobs¶
Exercise 1¶
Captured 2026-04-15T01:48:00Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --partition=gpu --gres=gpu:1 --cpus-per-task=4 --mem=8G --time=00:10:00 --job-name=val_gpu1 --output=gpu_%j.out --wrap='echo CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES; nvidia-smi --query-gpu=name,memory.total --format=csv'
Submitted batch job 42
$ cat gpu_42.out
CUDA_VISIBLE_DEVICES=0
name, memory.total [MiB]
Tesla T4, 15360 MiB
Exercise 2¶
Captured 2026-04-15T01:50:27Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 3¶
Captured 2026-04-15T01:50:27Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --partition=gpu --gres=gpu:1 --cpus-per-task=4 --mem=8G --time=00:10:00 --job-name=val_gpu3 --output=gpu3_%j.out --wrap='echo CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES; nvidia-smi --query-gpu=index,name,memory.total --format=csv'
Submitted batch job 42
$ cat gpu3_42.out
CUDA_VISIBLE_DEVICES=0
index, name, memory.total [MiB]
0, Tesla T4, 15360 MiB
Exercise 4¶
Captured 2026-04-15T01:50:48Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sacct -j 42 --format=JobID,JobName,AllocTRES%60,Elapsed,State
JobID JobName AllocTRES Elapsed State
------------ ---------- ------------------------------------------------------------ ---------- ----------
42 val_gpu4 billing=4,cpu=4,mem=8G,node=1 00:00:10 COMPLETED
42.batch batch cpu=4,mem=8G,node=1 00:00:10 COMPLETED
user/interactive-jobs¶
Exercise 1¶
Captured 2026-04-15T01:50:49Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ srun --cpus-per-task=2 --mem=4G --time=00:05:00 hostname
batch-dy-compute-1
$ srun --cpus-per-task=2 --mem=4G --time=00:05:00 nproc
2
Exercise 2¶
Captured 2026-04-15T01:50:50Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ srun --cpus-per-task=4 --mem=8G --time=00:05:00 hostname
batch-dy-compute-1
$ srun --cpus-per-task=4 --mem=8G --time=00:05:00 free -m | head -2
total used free shared buff/cache available
Mem: 31720 481 30540 10 698 30837
Exercise 4¶
Captured 2026-04-15T01:50:50Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ srun --time=00:05:00 uname -a
Linux batch-dy-compute-1 6.12.73-95.123.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Feb 24 23:31:49 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
user/job-arrays¶
Exercise 1¶
Captured 2026-04-15T01:50:52Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 2¶
Captured 2026-04-15T01:50:57Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 3¶
Captured 2026-04-15T01:51:02Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --array=1-10%3 --time=00:05:00 --mem=256M --job-name=val_ja3 --output=/dev/null --wrap='sleep 30'
Submitted batch job 42
$ squeue -j 42 -h -t RUNNING -o '%i' | wc -l
3
Exercise 4¶
Captured 2026-04-15T01:51:17Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scancel 42_3
$ squeue -j 42 -r -o '%A_%a %T'
JOBID_ACCOUNT STATE
42_pcdefault RUNNING
574_pcdefault RUNNING
572_pcdefault RUNNING
571_pcdefault RUNNING
Exercise 5¶
Captured 2026-04-15T01:51:20Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
user/job-dependencies¶
Exercise 1¶
Captured 2026-04-15T01:51:26Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ squeue --me -o '%.8i %.20j %.2t %E'
JOBID NAME ST DEPENDENCY
580 val_jd1b PD afterok:579(unfulfilled)
579 val_jd1a PD (null)
Exercise 2¶
Captured 2026-04-15T01:51:31Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ squeue --me -o '%.8i %.20j %.2t %E'
JOBID NAME ST DEPENDENCY
583 val_jd2c PD afterok:581(unfulfilled),afterok:582(unfulfilled)
582 val_jd2b PD (null)
581 val_jd2a PD (null)
Exercise 3¶
Captured 2026-04-15T01:51:44Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ squeue --me -o '%.8i %.20j %.2t %R'
JOBID NAME ST NODELIST(REASON)
585 val_jd3_sg PD (Dependency)
584 val_jd3_sg R batch-dy-compute-1
Exercise 4¶
Captured 2026-04-15T01:51:54Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sacct -j 42,43,44 --format=JobID,JobName,State,ExitCode -X
JobID JobName State ExitCode
------------ ---------- ---------- --------
42 val_jd4f FAILED 1:0
43 val_jd4n PENDING 0:0
44 val_jd4h COMPLETED 0:0
$ squeue -j 43 -o '%.8i %.20j %.2t %.30r'
JOBID NAME ST REASON
43 val_jd4n PD DependencyNeverSatisfied
user/managing-jobs¶
Exercise 1¶
Captured 2026-04-15T01:51:55Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --time=01:00:00 --job-name=val_mg1 --output=/dev/null --wrap='sleep 3600'
Submitted batch job 42
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
42 batch val_mg1 ec2-user R 0:02 1 batch-dy-compute-1
$ scancel 42
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
Exercise 2¶
Captured 2026-04-15T01:51:59Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --hold --time=00:10:00 --job-name=val_mg2 --output=/dev/null --wrap='echo released'
Submitted batch job 42
$ squeue --me -o '%.8i %.20j %.2t %R'
JOBID NAME ST NODELIST(REASON)
42 val_mg2 PD (JobHeldUser)
$ scontrol release 42
$ squeue --me -o '%.8i %.20j %.2t %R'
JOBID NAME ST NODELIST(REASON)
Exercise 3¶
Captured 2026-04-15T01:52:04Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --hold --time=01:00:00 --job-name=val_mg3 --output=/dev/null --wrap='sleep 60'
Submitted batch job 42
$ scontrol update JobId=42 TimeLimit=02:00:00
$ scontrol show job 42 | grep TimeLimit
RunTime=00:00:00 TimeLimit=02:00:00 TimeMin=N/A
$ scontrol release 42
Exercise 4¶
Captured 2026-04-15T01:52:06Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --time=00:05:00 --job-name=val_mg4a --output=/dev/null --wrap='sleep 60'
Submitted batch job 42
$ sbatch --time=00:05:00 --job-name=val_mg4b --output=/dev/null --wrap='sleep 60'
Submitted batch job 43
$ sbatch --hold --time=00:05:00 --job-name=val_mg4p --output=/dev/null --wrap='sleep 60'
Submitted batch job 44
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
44 batch val_mg4p ec2-user PD 0:00 1 (JobHeldUser)
42 batch val_mg4a ec2-user R 0:01 1 batch-dy-compute-1
43 batch val_mg4b ec2-user R 0:01 1 batch-dy-compute-2
$ scancel --me --state=PENDING
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
42 batch val_mg4a ec2-user R 0:03 1 batch-dy-compute-1
43 batch val_mg4b ec2-user R 0:03 1 batch-dy-compute-2
$ scancel --me
user/monitoring-jobs¶
Exercise 1¶
Captured 2026-04-15T01:52:12Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ squeue --me -o '%.8i %.9P %.30j %.2t %.10M %.10l %R'
JOBID PARTITION NAME ST TIME TIME_LIMIT NODELIST(REASON)
592 batch val_mg4a CG 0:03 5:00 batch-dy-compute-1
593 batch val_mg4b CG 0:03 5:00 batch-dy-compute-2
595 batch val_mj1_format PD 0:00 10:00 (None)
Exercise 2¶
Captured 2026-04-15T01:52:14Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scontrol show job 42
JobId=42 JobName=val_mj2_inspect
UserId=ec2-user(1000) GroupId=ec2-user(1000) MCS_label=N/A
Priority=3500 Nice=0 Account=pcdefault QOS=normal
JobState=CONFIGURING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:02 TimeLimit=00:10:00 TimeMin=N/A
SubmitTime=2026-04-15T01:52:12 EligibleTime=2026-04-15T01:52:12
AccrueTime=2026-04-15T01:52:12
StartTime=2026-04-15T01:52:12 EndTime=2026-04-15T02:02:12 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-15T01:52:12 Scheduler=Main
Partition=batch AllocNode:Sid=ip-172-31-94-160:186352
ReqNodeList=(null) ExcNodeList=(null)
NodeList=batch-dy-compute-3
BatchHost=batch-dy-compute-3
NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=2 ReqB:S:C:T=0:0:*:*
ReqTRES=cpu=2,mem=2G,node=1,billing=2
AllocTRES=cpu=2,mem=2G,node=1,billing=2
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=2 MinMemoryNode=2G MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null)
Command=(null)
SubmitLine=sbatch --time=00:10:00 --mem=2G --cpus-per-task=2 --wrap=sleep 300 --job-name=val_mj2_inspect
WorkDir=/home/ec2-user/slurm-val-1eVioE
StdErr=
StdIn=/dev/null
StdOut=/home/ec2-user/slurm-val-1eVioE/slurm-42.out
TresPerTask=cpu=2
Exercise 3¶
Captured 2026-04-15T01:52:24Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sacct -j 42 --format=JobID,JobName,ReqMem,MaxRSS,State
JobID JobName ReqMem MaxRSS State
------------ ---------- ---------- ---------- ----------
42 val_mj3 512M COMPLETED
42.batch batch 844K COMPLETED
Exercise 4¶
Captured 2026-04-15T01:52:24Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sinfo -p batch -t idle -o '%n %c %m'
HOSTNAMES CPUS MEMORY
batch-dy-compute-3 8 31129
batch-dy-compute-4 8 31129
batch-dy-compute-1 8 31129
batch-dy-compute-2 8 31129
Exercise 5¶
Captured 2026-04-15T01:52:27Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ squeue --me --start
JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON)
598 batch val_mj5 ec2-user PD N/A 1 (null) (JobHeldUser)
user/parallel-mpi-jobs¶
Exercise 1¶
Captured 2026-04-15T01:52:28Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 2¶
Captured 2026-04-15T01:52:33Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 3¶
Captured 2026-04-15T01:52:39Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 4¶
Captured 2026-04-15T01:52:44Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
user/recurring-jobs-scrontab¶
Exercise 1¶
Captured 2026-04-15T01:52:50Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 2¶
Captured 2026-04-15T01:52:50Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ scrontab -l
#SCRON --time=00:10:00
#SCRON --mem=1G
#SCRON -J val_scron_hourly
#SCRON --output=/home/ec2-user/logs/val_scron_hourly_%j.out
@hourly /home/ec2-user/scripts/val_scron_hourly.sh
$ squeue --me --name=val_scron_hourly || true
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
604 batch val_scro ec2-user PD 0:00 1 (BeginTime)
Exercise 3¶
Captured 2026-04-15T01:52:50Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
user/resource-requests¶
Exercise 1¶
Captured 2026-04-15T01:52:51Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 2¶
Captured 2026-04-15T01:52:56Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --exclusive --mem=0 --time=00:10:00 --job-name=val_rr2 --output=exclusive_%j.out --wrap='echo CPUs: $(nproc); free -g'
Submitted batch job 42
Exercise 3¶
Captured 2026-04-15T01:53:01Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sinfo -o '%n %f'
HOSTNAMES AVAIL_FEATURES
batch-dy-compute-3 dynamic,t3.2xlarge,compute
batch-dy-compute-4 dynamic,t3.2xlarge,compute
batch-dy-compute-1 dynamic,t3.2xlarge,compute
batch-dy-compute-2 dynamic,t3.2xlarge,compute
debug-dy-small-1 dynamic,t3.medium,small
debug-dy-small-2 dynamic,t3.medium,small
gpu-dy-t4-1 dynamic,g4dn.xlarge,t4,gpu
highmem-dy-large-1 dynamic,t3.2xlarge,large
Exercise 4¶
Captured 2026-04-15T01:53:01Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --time=00:05:00 --mem=16G --job-name=val_rr4 --output=/dev/null --wrap='sleep 10'
Submitted batch job 42
$ sacct -j 42 --format=JobID,JobName,ReqMem,MaxRSS,Elapsed,State
JobID JobName ReqMem MaxRSS Elapsed State
------------ ---------- ---------- ---------- ---------- ----------
42 val_rr4 16G 00:00:11 COMPLETED
42.batch batch 900K 00:00:11 COMPLETED
Exercise 5¶
Captured 2026-04-15T01:53:17Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --parsable --hold --time=00:05:00 --mem=16G --cpus-per-task=4 --job-name=val_rr5a --wrap='sleep 60'
608
$ sbatch --parsable --hold --time=00:05:00 --mem-per-cpu=4G --cpus-per-task=4 --job-name=val_rr5b --wrap='sleep 60'
609
$ scontrol show job 608 | grep -E '(MinMemory|TRES=)'
ReqTRES=cpu=4,mem=16G,node=1,billing=4
AllocTRES=(null)
MinCPUsNode=4 MinMemoryNode=16G MinTmpDiskNode=0
$ scontrol show job 609 | grep -E '(MinMemory|TRES=)'
ReqTRES=cpu=4,mem=16G,node=1,billing=4
AllocTRES=(null)
MinCPUsNode=4 MinMemoryCPU=4G MinTmpDiskNode=0
$ scancel 608 609
user/slurm-overview¶
Exercise 1¶
Captured 2026-04-15T01:53:20Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up infinite 2 idle~ batch-dy-compute-[3-4]
batch* up infinite 2 idle batch-dy-compute-[1-2]
debug up infinite 2 idle~ debug-dy-small-[1-2]
gpu up infinite 1 idle gpu-dy-t4-1
highmem up infinite 1 idle~ highmem-dy-large-1
$ sinfo --summarize
PARTITION AVAIL TIMELIMIT NODES(A/I/O/T) NODELIST
batch* up infinite 0/4/0/4 batch-dy-compute-[1-4]
debug up infinite 0/2/0/2 debug-dy-small-[1-2]
gpu up infinite 0/1/0/1 gpu-dy-t4-1
highmem up infinite 0/1/0/1 highmem-dy-large-1
Exercise 2¶
Captured 2026-04-15T01:53:20Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
user/submitting-jobs¶
Exercise 1¶
Captured 2026-04-15T01:53:21Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 2¶
Captured 2026-04-15T01:53:26Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --time=00:05:00 --mem=1G --job-name=val_sj2 --output=/dev/null --wrap='date; head -5 /proc/cpuinfo'
Submitted batch job 42
Exercise 3¶
Captured 2026-04-15T01:53:26Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
$ sbatch --mail-type=BEGIN,END,FAIL --mail-user=you@example.com --time=00:05:00 --job-name=val_sj3 --wrap='sleep 30'
Submitted batch job 42
$ scontrol show job 42 | grep -i mail
SubmitLine=sbatch --mail-type=BEGIN,END,FAIL --mail-user=you@example.com --time=00:05:00 --job-name=val_sj3 --wrap=sleep 30
MailUser=you@example.com MailType=BEGIN,END,FAIL
Exercise 4¶
Captured 2026-04-15T01:53:28Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0
Exercise 5¶
Captured 2026-04-15T01:53:33Z · cluster slurm-training-val · Slurm 25.11.4 · ParallelCluster 3.15.0