Bureaucrats, cc_docs_admin, cc_staff
2,232
edits
Storage and file management: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
No edit summary |
||
| Line 37: | Line 37: | ||
* As far as possible, use scratch and local storage for temporary files. For local storage you can use the temporary directory created by the [[Running jobs|job scheduler]] for this, named <code>$SLURM_TMPDIR</code>. | * As far as possible, use scratch and local storage for temporary files. For local storage you can use the temporary directory created by the [[Running jobs|job scheduler]] for this, named <code>$SLURM_TMPDIR</code>. | ||
* If your program must search within a file, it is fastest to do it by first reading it completely before searching. | * If your program must search within a file, it is fastest to do it by first reading it completely before searching. | ||
* If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible | * If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible move them to an alternative location like [[Using nearline storage|nearline]]. | ||
* For more notes on managing many files, see [[Handling large collections of files]], especially if you are limited by a quota on the number of files. | * For more notes on managing many files, see [[Handling large collections of files]], especially if you are limited by a quota on the number of files. | ||
* Having any sort of parallel write access to a file stored on a shared filesystem like home, scratch and project is likely to create problems unless you are using a specialized tool such as [https://en.wikipedia.org/wiki/Message_Passing_Interface#I/O MPI-IO]. | * Having any sort of parallel write access to a file stored on a shared filesystem like home, scratch and project is likely to create problems unless you are using a specialized tool such as [https://en.wikipedia.org/wiki/Message_Passing_Interface#I/O MPI-IO]. | ||