cc_staff
411
edits
Storage and file management: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
No edit summary |
||
| Line 25: | Line 25: | ||
Project (def-username) 4096 B/9536 GB 2/500k | Project (def-username) 4096 B/9536 GB 2/500k | ||
</pre> | </pre> | ||
== Storage types == <!--T:5--> | == Storage types == <!--T:5--> | ||
| Line 38: | Line 32: | ||
* '''SCRATCH''': For intensive read/write operations on large files (> 100 MB per file), scratch is the best choice. Remember however that important files must be copied off scratch since they are not backed up there, and older files are subject to [[Scratch purging policy|purging]]. The scratch storage should therefore be used for temporary files: checkpoint files, output from jobs and other data that can easily be recreated. | * '''SCRATCH''': For intensive read/write operations on large files (> 100 MB per file), scratch is the best choice. Remember however that important files must be copied off scratch since they are not backed up there, and older files are subject to [[Scratch purging policy|purging]]. The scratch storage should therefore be used for temporary files: checkpoint files, output from jobs and other data that can easily be recreated. | ||
* '''SLURM_TMPDIR''': While a job is running, <code>$SLURM_TMPDIR</code> is a unique path to a temporary folder on a local fast filesystem on each compute node reserved for the job. This is the best location to temporarily store large collections of small files (< 1 MB per file). Note: this space is shared between jobs on each node, and the total available space depends on the node specifications. Finally, when the job ends, this folder is deleted. | * '''SLURM_TMPDIR''': While a job is running, <code>$SLURM_TMPDIR</code> is a unique path to a temporary folder on a local fast filesystem on each compute node reserved for the job. This is the best location to temporarily store large collections of small files (< 1 MB per file). Note: this space is shared between jobs on each node, and the total available space depends on the node specifications. Finally, when the job ends, this folder is deleted. | ||
==Breakdown of storage usage per user== | |||
While the command '''diskusage_report''' gives the space and inode usage per user on ''home'' and ''scratch'', it shows the total usage of the group on project. Since the files that belong to a given user could be all over the place under project, it is difficult to get a break down per user and per given project in case a user has access to more than one project. However, a user can get an estimation of his space and inode usage on the entire project file system by running the command: | |||
<code>lfs quota -u $USER /project</code> | |||
== Best practices == <!--T:9--> | == Best practices == <!--T:9--> | ||