WestGrid Legacy Systems Migration
- 1 WestGrid Legacy Systems Migration Process 2018-2019
- 2 Data Retention and Deletion Policy for Legacy Systems
- 3 Where to migrate to?
- 4 Code and Job Migration
- 5 How to migrate data?
- 6 Orcinus (University of British Columbia)
- 7 Support & Other Links
- 8 Previously Defunded Systems Notes
- 9 Document History
WestGrid Legacy Systems Migration Process 2018-2019
The Canada Foundation for Innovation (CFICanada Foundation for Innovation) has agreed to continue funding Orcinus as a Compute Canada / WestGrid system for one more year. It will be decommissioned in 2019.
|System||Site||Defunding Date||Recommended Migration Date(*)||Notes|
||University of British Columbia|| *updated*
| For more information on this system, read below. |
Non-RAC users are responsible for their own data.
To begin preparing your files, read these General Directives for Migration.
“Defunded” means that the system is no longer funded, operated or maintained as part of Compute Canada’s national cyberinfrastructure platform. For defunded systems, the host institution assumes full control of the system after the defunding date, including managing the storage file systems and backups. Users should contact Local Site Support of defunded systems for further information about the host institution’s data retention and deletion policies:
Users on the above systems will need to migrate their data and jobs to the new systems before the defunding date.
Please email firstname.lastname@example.org to request help with moving data or any other concerns with this migration policy.
Data Retention and Deletion Policy for Legacy Systems
User data on defunded systems will be deleted. Users are responsible for migrating their data to alternate storage.
IMPORTANT: For defunded systems, the host institution assumes full control of the system, including managing the storage systems and backups. Users should contact Local Site Support of defunded systems for further information about the host institution’s data retention and deletion policies.
Data on defunded systems will be deleted after the published deletion dates. WestGrid has arranged with the host institutions to keep data until the published deletion date, but can make no guarantees about data retention after that date. WestGrid will not retain any long term or back-up copies of user data and as noted above users must arrange for migration of their data. Users should also ensure they take the appropriate steps to comply with any data management requirements their institution or project may require.
Where to migrate to?
Users with RAC 2017 (current) Allocations
We have a conflict here in that users with current allocations will naturally want to use their allocations and associated priority right to the end of the allocation year (Mar.31). But users also have to migrate applications and especially data before the defunding date (also Mar.31).
Hopefully users will be able to do both:
- Continue to submit jobs on the legacy systems right to the end.
- In parallel migrate applications and data to a new system in preparation for the final move shortly before the defunding date.
The National_Data_Cyberinfrastructure (NDC) provides a backed-up, reliable storage system (the
/project space) mounted on each of Cedar and Graham. So generally users should copy their data to one of those two systems.
A few special notes:
- If you have a RAC 2018 award you may have been allocated to Cedar or Graham, therefore you should choose the PROJECT storage attached to that system;
- If you do not have a RAC 2018 award then you may use Rapid Access Service (RAS) storage amounts at either SFU or Waterloo at your discretion.
See the following useful pages:
The Niagara large parallel system in Toronto is currently (Dec.11/2017) being delivered and is being allocated for RAC 2018. We expect the system to become available later in the winter. Those of you with large parallel requirements may be allocated to Niagara and will be notified as usual through the RAC process (mid-March).
Users Applying for RAC 2018 Awards
All RAC 2018 allocations will be implemented on the new systems. We will do our best to satisfy specific requests but may have to juggle a few awards to ensure reasonably consistent usage.
Software Available on New Systems
Software lists have been compiled and installation scripts developed so that software can be (mostly) automatically installed in Cedar and Graham as soon as they are available. The software list is continuously updated: current list of available software.
Some commercial software will be licensed nationally. This is still under discussion within Compute Canada. Please check back for updates.
Code and Job Migration
Please keep in mind that you may need to re-compile or re-install your software and any required packages on new systems. This can be a time-consuming process. Our support staff have considerable expertise in such tasks, so please feel free to contact email@example.com for help.
See Code and job migration_from legacy systems for various details that should be kept in mind.
How to migrate data?
User Responsibility for Data Migration
We would like to emphasize that each user is responsible for copying his/her data from the to-be-defunded systems. WestGrid has no ability to track individual data transfers so we cannot send out individual reminders, and the defunding dates are hard deadlines. You must copy your data to a new, reliable site well before the defunding date. WestGrid recommends starting at least 3 months before the defunding date.
General File Management Best Practices and Suggestions
- Users must move their own data to a new system BEFORE the defunding dates.
- Delete any unnecessary data and files
- Refrain from keeping multiple copies of your data on multiple systems.
- Move any remaining data not currently being used to a long term storage site.
- Learn about the PROJECT (backed-up) and NEARLINE (tape only) services available on the National Data Cyberinfrastructure.
- Scratch storage is not backed up, and has no long-term availability guarantees. Inactive data in scratch storage areas is subject to being purged. We have noticed that some users seem to be storing more important data on scratch. Therefore please check your scratch storage and move anything important to permanent storage.
Recommendations for Large Numbers of Files
We have a few users with very large numbers of files (many thousands). It is notoriously inefficient to transfer such a large number of files with the usual file-transfer utilities. We strongly recommend that users consider
taring up their files before transfer. See HERE for an (external) tutorial on how to compress and extract files using the tar command on Linux or the much more detailed discussion and examples in the Compute Canada Wiki at archiving_and_compressing_files.
GlobusGlobus is a file transfer service [https://www.globus.org/] for Data Transfer
Globus is the ideal tool as it significantly improves transfer performance and reduces the time spent managing transfers. Users have reported 10x or even 100x improvements over other transfer methods such as SCP.
GlobusGlobus is a file transfer service [https://www.globus.org/] can be used to move data between any two WestGrid resources. All WestGrid resources are already configured as GlobusGlobus is a file transfer service [https://www.globus.org/] endpoints.
CLICK HERE for more info on using GlobusGlobus is a file transfer service [https://www.globus.org/] File Transfer. Refer to the Best practices for data migration page for detailed instructions on moving your data to a new system.
Owncloud and Database Services
WestGrid provides a Database service at SFU. It is not scheduled for defunding. New database servers will be installed and WestGrid/SFU will contact users and help them move all data. No action is required at this time.
Similarly for Owncloud. This is a separate service which is not scheduled for defunding. The service will be moved to new Compute Canada servers at which time WestGrid/SFU will contact users and help them move their data. No action is required at this time.
Orcinus (University of British Columbia)
UPDATE: Continuation to MARCH 31, 2019
The Canada Foundation for Innovation (CFICanada Foundation for Innovation) has agreed to continue funding Orcinus as a Compute Canada / WestGrid system until Mar 31, 2019. WestGrid very much appreciates this support as the demand for access to advanced research computing resources continues to grow each year across the country.
Orcinus has a small attached disc system providing
/globalscratch. Users should migrate
/home to either Cedar or Graham. The Orcinus
/globalscratch is not backed up and was aimed at providing high performance scratch. However we have noticed that some users have what looks like important or permanent data on
/globalscratch. Please migrate any such data to Cedar or Graham.
Orcinus is a general purpose system used for both serial and parallel computation. Any of the new general purpose systems would be suitable for typical orcinus users, and the new large parallel system Niagara will be available for RAC 2018 (mid-April 2018).
Support & Other Links
Check out the following links for more tools, tips and support related to migration:
- Frequently Asked Questions
- Recommendations for Migration to New Systems
- Rapid Access Service (opportunistic use)
Previously Defunded Systems Notes
|System||Site||Defunding Date||Recommended Migration Date(*)||Notes|
||University of Victoria||June 1, 2017||(COMPLETE)||
UVic may keep
||University of Calgary||August 31, 2017||(COMPLETE)|
||University of Alberta||September 30, 2017||(COMPLETE)||
UofA is intending to keep jasper and hungabee going. For details please ask local IT support.
||University of British Columbia||March 31, 2018||(COMPLETE)||
If you have questions, contact firstname.lastname@example.org.
||University of Manitoba||March 31, 2018||(COMPLETE)||
The new large parallel system Niagara will be available for the RAC 2018 year and would be an excellent alternative for those users with large parallel ambitions. WebMO is available on the ACENET Placentia system and Gaussian is available on Cedar. If you have questions, contact email@example.com.
||University of Calgary||March 31, 2018||(COMPLETE)||
Local Users (any researcher located at or collaborating with the University of Calgary) should email firstname.lastname@example.org to clarify local use policies or to confirm if you qualify as a local user.
|v2.0||Apr. 03, 2018||Removed information for systems defunded on March 31, 2018 (Parallel, Orcinus, Grex) and updated the Previously Defunded Systems Notes table.|
|v1.9||Feb. 02, 2018||Added links to General Directives for Migration for users on remaining legacy systems (Parallel, Orcinus, Grex).|
|v1.8||Dec.11, 2017||Bugaboo defunding delayed. Removed details for previously defunded systems.|
|v1.7||Oct.19, 2017||Bugaboo migration and defunding plans updated.|
|v1.6||Sep.12, 2017||2018 migration and defunding plans added.|
|v1.5||Aug 22, 2017||Some new systems are now available and in production. The text has been updated to reflect actuals.|
|v1.4||June 30, 2017||RAC 2017 allocations have been implemented on the continuing WG legacy systems.|
|v1.3||May 31, 2017||Schedule updates and various details for systems to be defunded in 2017.|
|v1.2||Mar.20, 2017||Link to software availability documentation|
|v1.1||Mar.9, 2017||Nestor has been extended to June 1, 2017.|
|v1.0||Mar.8, 2017||UofC dates revised to Aug.31. General clean-up of the docs.|
|v0.92||Mar.7, 2017||Revised dates. University of Alberta details updated. User access to Jasper & Hungabee extended to September 30, 2017.|
|v0.91||Mar 3, 2017||UVic nestor/hermes details updated and confirmed (to June 1, 2017)|