Bureaucrats, cc_docs_admin, cc_staff
2,774
edits
General directives for migration: Difference between revisions
Jump to navigation
Jump to search
→Archive and compress
| Line 20: | Line 20: | ||
Most file transfer programs move one file of a reasonable size more efficiently than thousands of small files of equal total size. If you have directories or directory trees containing many small files, use [[Archiving and compressing files|tar]] to combine (archive) them. | Most file transfer programs move one file of a reasonable size more efficiently than thousands of small files of equal total size. If you have directories or directory trees containing many small files, use [[Archiving and compressing files|tar]] to combine (archive) them. | ||
Large files can | Large files can benefit from compression in some cases, especially text files which can usually be compressed a great deal. Compressing a file ''only'' for the purpose of transferring it, and then decompressing it at the end of the transfer, will not necessarily save time though. It depends on how small the file can be compressed, how long it takes to compress it, and the transfer bandwidth. The calculation is described in the "Data Compression and transfer discussion" of [https://bluewaters.ncsa.illinois.edu/data-transfer-doc this document] from the US National Center for Supercomputing Applications. | ||
If you decide compression is worthwhile you can use again use [[Archiving and compressing files|tar]] for this, or [https://www.gnu.org/software/gzip/manual/gzip.html gzip]. | |||
=== Avoid duplication === | === Avoid duplication === | ||