Cedar: Difference between revisions

Jump to navigation Jump to search

Cedar (view source)

Revision as of 17:09, 19 June 2018

265 bytes removed ,  5 years ago
reformat hardware table
(Usable memory changed to preferred Slurm usage)
(reformat hardware table)
Line 66: Line 66:


<!--T:17-->
<!--T:17-->
Cedar has a total of 58,416 CPU cores for computation, and 584 GPU devices. Total theoretical peak double precision performance is 936 teraflops for CPUs, plus 2,744 for GPUs, yielding over 3.6 petaflops of theoretical peak double precision performance. 22 fully connected "islands" of 32 base or large nodes each have 1024 cores in a fully non-blocking topology (Omni-Path fabric), with each island designed to yield over 30 teraflops of double-precision performance (measured with high performance LINPACK). There is a 2:1 blocking factor between the 1024 core islands.
Cedar has a total of 58,416 CPU cores for computation, and 584 GPU devices. Total theoretical peak double precision performance is 936 teraflops for CPUs, plus 2,744 for GPUs, yielding over 3.6 petaflops of theoretical peak double precision performance. 22 fully connected "islands" of 32 base or large nodes each have 1024 cores in a fully non-blocking topology (Omni-Path fabric), with each island designed to yield over 30 teraflops of double-precision performance (measured with high performance LINPACK). There is a 2:1 blocking factor between the 1024 core islands.  


<!--T:7-->
<!--T:7-->
{| class="wikitable sortable"
{| class="wikitable sortable"
! Count !! Node type !! Cores !! Available memory !! Hardware detail
|-
|-
| base nodes || 576 nodes || 131072M of memory (128000M usable, 4000M per core), 16 cores/socket, 2 sockets/node.  Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4.
| 576 || base "128G"    || 32 || 125G or 128000M   || two Intel E5-2683 v4 "Broadwell" at 2.1Ghz
|-
|-
| large nodes || 128 nodes || 262144M of memory (256000M usable, 8000M per core), 16 cores/socket, 2 sockets/node.  Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4.
| 128 || large "256G"    || 32 || 250G or 257000M  || (same as base nodes)
|-
|-
| GPU base nodes || 114 nodes || 131072M of memory (128000M usable, 4000M per core), 12 cores/socket, 2 sockets/node, 4 NVIDIA P100 Pascal GPUs/node (12GB HBM2 memory), 2 GPUs/PCI root. Intel "Broadwell" CPUs at 2.2Ghz, model E5-2650 v4
| 24  || large "512G"    || 32 || 502G or 515000M  || (same as base nodes)
|-
|-
| GPU large nodes || 32 nodes || 262144M of memory (256000M usable, 8000M per core), 12 cores/socket, 2 sockets/node, 4 NVIDIA P100 Pascal GPUs/node (16GB HBM2 memory), All GPUs on the same PCI root.  E5-2650 v4
| 24 ||bigmem1500 "1.5T"|| 32 || 1510G or 1547000M || (same as base nodes)
|-
|-
| bigmem500 nodes || 24 nodes || 524288M of memory (512000M usable, 16000M per core), 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4.
| 4  || bigmem3000 "3T" || 64 || 3022G or 3095000M || four Intel E7-4809 v4 "Broadwell" at 2.1Ghz
|-
|-
|bigmem1500 nodes || 24 nodes || 1572864M (1.5TiB) of memory (1536000M usable, 48000M per core), 16 cores/socket, 2 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E5-2683 v4.
| 114 || base GPU        || 24 || 125G or 128000M  || two E5-2650 v4 at 2.2GHz + four NVIDIA P100 Pascal GPUs (12GB HBM2 memory)
|-
|-
| bigmem3000 nodes || 4 nodes || 3145728M (3TiB) of memory (3072000M usable, 96000M per core), 8 cores/socket, 4 sockets/node. Intel "Broadwell" CPUs at 2.1Ghz, model E7-4809 v4.
| 32  || large GPU      || 24 || 250G or 257000M  || two E5-2650 v4 at 2.2GHz + four NVIDIA P100 Pascal GPUs (16GB HBM2 memory)
|-
|-
| Skylake base nodes || 640 nodes || 196608M of memory (192000M usable, 4000M per core), 24 cores/socket, 2 sockets/node. Intel "Skylake" CPUs at 2.1Ghz, model Platinum 8160F.
| 640 || Skylake         || 48 || 187G or 192000M   || two Intel Platinum 8160F "Skylake" at 2.1Ghz
|}
|}
Note that the amount of available memory is less than the "round number" suggested by the hardware configuration. For instance, "base" nodes do have 128 GiB of RAM, but some of it is permanently occupied by the kernel and OS. To avoid wasting time by swapping/paging, the scheduler will never allocate jobs whose memory requirements exceed the amount of "available" memory shown above.


<!--T:10-->
<!--T:10-->
All of the above nodes have local (on-node) temporary storage. GPU nodes have a single 800GB SSD drive. All other compute nodes have two 480GB SSD drives, for a total raw capacity of 960GB. Best practice to access node-local storage is to use the directory generated by [[Running jobs|Slurm]], $SLURM_TMPDIR.
All nodes have local (on-node) temporary storage. GPU nodes have a single 800GB SSD drive. All other compute nodes have two 480GB SSD drives, for a total raw capacity of 960GB. Best practice to access node-local storage is to use the directory generated by [[Running jobs|Slurm]], $SLURM_TMPDIR.


<!--T:18-->
<!--T:18-->
Rdickson
Bureaucrats, cc_docs_admin, cc_staff
2,773

edits

Retrieved from "https://docs.alliancecan.ca/wiki/Special:MobileDiff/53770"

Navigation menu