From CC Doc
Jump to: navigation, search
Other languages:
English • ‎français

Availability: Compute RAC2017 allocations started June 30, 2017
Login node: cedar.computecanada.ca
GlobusGlobus is a file transfer service [https://www.globus.org/] endpoint: computecanada#cedar-dtn
System Status Page: https://www.westgrid.ca/support/system_status

Cedar is a heterogeneous cluster suitable for a variety of workloads; it is located at Simon Fraser University. It is named for the Western Red Cedar, B.C.’s official tree, which is of great spiritual significance to the region's First Nations people. It was previously known as "GP2" and is still identified as such in the 2017 RAC documentation.

Cedar is sold and supported by Scalar Decisions, Inc. The node manufacturer is Dell, the high performance temporary space (scratch) is from DDN, and the interconnect is from Intel. It is entirely liquid cooled, using rear-door heat exchangers.

Getting started with Cedar

Attached storage

Home space
250TB total volume
  • Location of home directories.
  • Each home directory has a small fixed quota.
  • Not allocated via RAS or RAC. Larger requests go to Project space.
  • Has daily backup
Scratch space
3.7PB total volume
Parallel high-performance filesystem
  • For active or temporary (/scratch) storage.
  • Not allocated.
  • Large fixed quota per user.
  • Inactive data will be purged.
Project space
10PB total volume
External persistent storage

Scratch storage is a Lustre filesystem based on DDN model ES14K technology. It includes 640 8TB NL-SAS disk drives, and dual redundant metadata controllers with SSD-based storage.

High-performance interconnect

Intel OmniPath (version 1) interconnect (100Gbit/s bandwidth).

A low-latency high-performance fabric connecting all nodes and temporary storage.

By design, Cedar supports multiple simultaneous parallel jobs of up to 1024 broadwell cores (32 nodes) or 1536 skylake cores (32 nodes) in a fully non-blocking manner. For larger jobs the interconnect has a 2:1 blocking factor, i.e., even for jobs running on several thousand cores, Cedar provides a high-performance interconnect.

Node types and characteristics

Cedar has a total of 58,416 CPU cores for computation, and 584 GPU devices.

Count Node type Cores Available memory Hardware detail
576 base "128G" 32 125G or 128000M two Intel E5-2683 v4 "Broadwell" at 2.1Ghz
128 large "256G" 32 250G or 257000M (same as base nodes)
24 large "512G" 32 502G or 515000M (same as base nodes)
24 bigmem1500 "1.5T" 32 1510G or 1547000M (same as base nodes)
4 bigmem3000 "3T" 32 3022G or 3095000M four Intel E7-4809 v4 "Broadwell" at 2.1Ghz
114 base GPU 24 125G or 128000M two E5-2650 v4 at 2.2GHz + four NVIDIA P100 Pascal GPUs (12GB HBM2 memory)
32 large GPU 24 250G or 257000M two E5-2650 v4 at 2.2GHz + four NVIDIA P100 Pascal GPUs (16GB HBM2 memory)
640 Skylake 48 187G or 192000M two Intel Platinum 8160F "Skylake" at 2.1Ghz

Note that the amount of available memory is less than the "round number" suggested by the hardware configuration. For instance, "base" nodes do have 128 GiB of RAM, but some of it is permanently occupied by the kernel and OS. To avoid wasting time by swapping/paging, the scheduler will never allocate jobs whose memory requirements exceed the amount of "available" memory shown above.

All nodes have local (on-node) temporary storage. GPU nodes have a single 800GB SSD drive. All other compute nodes have two 480GB SSD drives, for a total raw capacity of 960GB. Best practice to access node-local storage is to use the directory generated by Slurm, $SLURM_TMPDIR.

Choosing a node type

Most applications will run on either Broadwell or Skylake nodes, and performance differences are expected to be small compared to job waiting times. Therefore we recommend that you do not select a specific node type for your jobs. If it is necessary, use --constraint=skylake or --constraint=broadwell. See Specifying a CPU architecture.


Theoretical peak double precision performance of Cedar is 936 teraflops for CPUs, plus 2,744 for GPUs, yielding over 3.6 petaflops of theoretical peak double precision performance. 22 fully connected "islands" of 32 base or large nodes each have 1024 cores in a fully non-blocking topology (Omni-Path fabric), with each island designed to yield over 30 teraflops of double-precision performance (measured with high performance LINPACK). There is a 2:1 blocking factor between the 1024 core islands. The Skylake nodes also span 20 non-blocking islands of 32 nodes each, forming islands of 1536 cores.