cc_staff
1,485
edits
General directives for migration: Difference between revisions
Jump to navigation
Jump to search
→What to do before the migration starts?
(simplify, shorten & copy-edit) |
|||
| Line 8: | Line 8: | ||
Make sure you know whether you are responsible for your own data migration, or whether Compute Canada staff will be migrating your data. Migration of certain legacy systems like [[Migration2016:Silo|Silo]] is being handled by staff. If you are in any doubt, write [mailto:support@computecanada.ca support@computecanada.ca]. | Make sure you know whether you are responsible for your own data migration, or whether Compute Canada staff will be migrating your data. Migration of certain legacy systems like [[Migration2016:Silo|Silo]] is being handled by staff. If you are in any doubt, write [mailto:support@computecanada.ca support@computecanada.ca]. | ||
If you haven't used [[Globus]] before, read about it now and verify that it works on the system you are migrating from. Test any other tools you will use (like [ | If you haven't used [[Globus]] before, read about it now and verify that it works on the system you are migrating from. Test any other tools you will use (like [http://www.howtogeek.com/248780/how-to-compress-and-extract-files-using-the-tar-command-on-linux/ tar], [https://www.gnu.org/software/gzip/manual/gzip.html gzip], [https://www.cyberciti.biz/faq/how-to-create-a-zip-file-in-unix/ zip]) on test data to ensure you know how they work before using them on important data. | ||
Do not wait until the last minute to start your migration. Depending on how much data you have and how much load there is on the machines and network, you may be surprised at how long it will take to finish a large transfer. Expect hundreds of gigabytes to take hours to transfer, but give yourself days in case there is a problem. Expect terabytes to take days. | Do not wait until the last minute to start your migration. Depending on how much data you have and how much load there is on the machines and network, you may be surprised at how long it will take to finish a large transfer. Expect hundreds of gigabytes to take hours to transfer, but give yourself days in case there is a problem. Expect terabytes to take days. | ||
| Line 18: | Line 18: | ||
=== Compress and archive === | === Compress and archive === | ||
Most file transfer programs move one file of a reasonable size more efficiently than thousands of small files of equal total size. If you have directories or directory trees containing many small files, use [ | Most file transfer programs move one file of a reasonable size more efficiently than thousands of small files of equal total size. If you have directories or directory trees containing many small files, use [http://www.howtogeek.com/248780/how-to-compress-and-extract-files-using-the-tar-command-on-linux/ tar] or [https://www.cyberciti.biz/faq/how-to-create-a-zip-file-in-unix/ zip] to combine (archive) and compress them. | ||
Large files can also benefit from compression in many cases, especially text files or numeric data stored as human-readable text. You can use again use [ | Large files can also benefit from compression in many cases, especially text files or numeric data stored as human-readable text. You can use again use [http://www.howtogeek.com/248780/how-to-compress-and-extract-files-using-the-tar-command-on-linux/ tar] for this, or [https://www.gnu.org/software/gzip/manual/gzip.html gzip], or [https://www.cyberciti.biz/faq/how-to-create-a-zip-file-in-unix/ zip]. | ||
=== Avoid duplication === | === Avoid duplication === | ||