rsnt_translations
53,770
edits
General directives for migration: Difference between revisions
Jump to navigation
Jump to search
General directives for migration (view source)
Revision as of 16:37, 27 November 2023
, 5 months agono edit summary
No edit summary |
No edit summary |
||
| Line 26: | Line 26: | ||
<!--T:8--> | <!--T:8--> | ||
Large files can benefit from compression in some cases, especially text files which can usually be compressed a great deal. Compressing a file | Large files can benefit from compression in some cases, especially text files which can usually be compressed a great deal. Compressing a file <b>only</b> for the purpose of transferring it, and then decompressing it at the end of the transfer, will not necessarily save time though. It depends on how much the file can be compressed, how long it takes to compress it, and the transfer bandwidth. The calculation is described under <i>Data Compression and transfer discussion</i> in [https://bluewaters.ncsa.illinois.edu/data-transfer-doc this document] from the US National Center for Supercomputing Applications. | ||
<!--T:9--> | <!--T:9--> | ||
If you decide compression is worthwhile you can again use [[Archiving and compressing files|tar]] for this, or [https://www.gnu.org/software/gzip/manual/gzip.html gzip]. | If you decide compression is worthwhile, you can again use [[Archiving and compressing files|tar]] for this, or [https://www.gnu.org/software/gzip/manual/gzip.html gzip]. | ||
=== Avoid duplication === <!--T:10--> | === Avoid duplication === <!--T:10--> | ||
| Line 38: | Line 38: | ||
== What to do during the migration process == <!--T:12--> | == What to do during the migration process == <!--T:12--> | ||
If it is supported at your source site, use [[Globus|Globus]] to set up your file transfer. It is the most user-friendly and efficient tool we know for this task. Globus is designed to recover from network interruptions automatically. We recommend you enable the setting to | If it is supported at your source site, use [[Globus|Globus]] to set up your file transfer. It is the most user-friendly and efficient tool we know for this task. Globus is designed to recover from network interruptions automatically. We recommend you enable the setting to <i>preserve source file modification times</i> in the <i>Transfer & Timer Options</i>. | ||
<!--T:13--> | <!--T:13--> | ||
If Globus is not supported at your source site, then compressing data and avoiding duplication is even more important. If you are using [https://en.wikipedia.org/wiki/Secure_copy scp], [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol sftp], or [https://en.wikipedia.org/wiki/Rsync rsync], then: | If Globus is not supported at your source site, then compressing data and avoiding duplication is even more important. If you are using [https://en.wikipedia.org/wiki/Secure_copy scp], [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol sftp], or [https://en.wikipedia.org/wiki/Rsync rsync], then: | ||
* Make a schedule to migrate your data in blocks of a few hundred GBs at a time. If the transfer stops for some reason, you will be able to try again starting from the incomplete file, but you will not have to re-transfer files that are already complete. An organized list of files will help here. | * Make a schedule to migrate your data in blocks of a few hundred GBs at a time. If the transfer stops for some reason, you will be able to try again starting from the incomplete file, but you will not have to re-transfer files that are already complete. An organized list of files will help here. | ||
* Check regularly to see that the transfer process has not stopped. File size is a good indicator of progress. If no files have changed size for several minutes, then something may have gone wrong. If restarting the transfer does not work, contact our [[ | * Check regularly to see that the transfer process has not stopped. File size is a good indicator of progress. If no files have changed size for several minutes, then something may have gone wrong. If restarting the transfer does not work, contact our [[technical support]]. | ||
<!--T:14--> | <!--T:14--> | ||