Bureaucrats, cc_docs_admin, cc_staff, rsnt_translations
2,807
edits
Known issues: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
No edit summary |
||
| Line 9: | Line 9: | ||
== Scheduler issues == <!--T:6--> | == Scheduler issues == <!--T:6--> | ||
* | * Exceeded step memory limit at some point | ||
** IO uses memory and slurm is correctly reporting these cases. | |||
** IO (and other things) can trigger an actual OOM kill, which triggers the same message (but affects exit status differently). | |||
** DerivedExitStatus 0:125 is the signature of the memory-use-but-not-OOM | |||
** in the absence of any other action, a step with 0:125 will *not* enable a dependent job which has afterok the latter is a slurm bug that will be fixed, so that slurm can clearly distinguish its handling of "out of memory" warning versus actual "kernel OOM killed" errors. Slurm will continue to correctly report memory usage from cgroups, so IO memory will still be counted. | |||
** This is a [https://bugs.schedmd.com/show_bug.cgi?id=3820 reported bug], which is reportedly fixed in an upcoming version of Slurm. | |||
* The CC Slurm configuration encourages whole-node jobs. When appropriate, users should request whole-node rather than per-core resources. See [[Job_scheduling_policies#Whole_nodes_versus_cores;|Job Scheduling - Whole Node Scheduling]]. | * The CC Slurm configuration encourages whole-node jobs. When appropriate, users should request whole-node rather than per-core resources. See [[Job_scheduling_policies#Whole_nodes_versus_cores;|Job Scheduling - Whole Node Scheduling]]. | ||
* By default, the job receives environment settings from the submitting shell. This can lead to irreproducible results if it's not what you expect. To force the job to run with a fresh-like login environment, you can submit with <tt>--export=none</tt> or add <tt>#SBATCH --export=NONE</tt> to your job script. | * By default, the job receives environment settings from the submitting shell. This can lead to irreproducible results if it's not what you expect. To force the job to run with a fresh-like login environment, you can submit with <tt>--export=none</tt> or add <tt>#SBATCH --export=NONE</tt> to your job script. | ||