rsnt_translations
53,770
edits
General directives for migration: Difference between revisions
Jump to navigation
Jump to search
General directives for migration (view source)
Revision as of 16:40, 27 November 2023
, 5 months agono edit summary
No edit summary |
No edit summary |
||
| Line 26: | Line 26: | ||
<!--T:8--> | <!--T:8--> | ||
Large files can benefit from compression in some cases, especially text files which can usually be compressed a great deal. Compressing a file <b>only</b> for the purpose of transferring it, and then decompressing it at the end of the transfer | Large files can benefit from compression in some cases, especially text files which can usually be compressed a great deal. Compressing a file <b>only</b> for the purpose of transferring it, and then decompressing it at the end of the transfer will not necessarily save time. It depends on how much the file can be compressed, how long it takes to compress it, and the transfer bandwidth. The calculation is described under <i>Data Compression and transfer discussion</i> in [https://bluewaters.ncsa.illinois.edu/data-transfer-doc this document] from the US National Center for Supercomputing Applications. | ||
<!--T:9--> | <!--T:9--> | ||
| Line 42: | Line 42: | ||
<!--T:13--> | <!--T:13--> | ||
If Globus is not supported at your source site, then compressing data and avoiding duplication is even more important. If you are using [https://en.wikipedia.org/wiki/Secure_copy scp], [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol sftp], or [https://en.wikipedia.org/wiki/Rsync rsync], then: | If Globus is not supported at your source site, then compressing data and avoiding duplication is even more important. If you are using [https://en.wikipedia.org/wiki/Secure_copy scp], [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol sftp], or [https://en.wikipedia.org/wiki/Rsync rsync], then: | ||
* Make a schedule to migrate your data in blocks of a few | * Make a schedule to migrate your data in blocks of a few hundreds of GBs at a time. If the transfer stops for some reason, you will be able to try again starting from the incomplete file, but you will not have to re-transfer files that are already complete. An organized list of files will help here. | ||
* Check regularly to see that the transfer process has not stopped. File size is a good indicator of progress. If no files have changed size for several minutes, then something may have gone wrong. If restarting the transfer does not work, contact our [[technical support]]. | * Check regularly to see that the transfer process has not stopped. File size is a good indicator of progress. If no files have changed size for several minutes, then something may have gone wrong. If restarting the transfer does not work, contact our [[technical support]]. | ||
| Line 49: | Line 49: | ||
== What to do after migration == <!--T:15--> | == What to do after migration == <!--T:15--> | ||
If you did not use Globus, or if you did but did not check | If you did not use Globus, or if you did but did not check <i>verify file integrity</i>, make sure that the data you have transferred are not corrupted. A crude way to do this is to compare file sizes at the source with file sizes at the destination. For greater assurance, you can use [http://man7.org/linux/man-pages/man1/cksum.1.html cksum] or [http://man7.org/linux/man-pages/man1/md5sum.1.html md5sum] at each end, and see if the results match. Any files with mismatching sizes or checksums should be transferred again. | ||
== Where and how to get help == <!--T:16--> | == Where and how to get help == <!--T:16--> | ||
* To know how to use different archiving and compression utilities, use a Linux command like <code>man <command></code> or <code><command> --help</code>. | * To know how to use different archiving and compression utilities, use a Linux command like <code>man <command></code> or <code><command> --help</code>. | ||
* Contact our [[ | * Contact our [[technical support]] | ||
</translate> | </translate> | ||