Bureaucrats, cc_docs_admin, cc_staff
2,773
edits
Cedar: Difference between revisions
Jump to navigation
Jump to search
relocate some paragraphs
(reformat hardware table) |
(relocate some paragraphs) |
||
| Line 51: | Line 51: | ||
* Has daily backup. | * Has daily backup. | ||
|} | |} | ||
<!--T:18--> | |||
Scratch storage is a Lustre filesystem based on DDN model ES14K technology. It includes 640 8TB NL-SAS disk drives, and dual redundant metadata controllers with SSD-based storage. | |||
=High-performance interconnect= <!--T:19--> | =High-performance interconnect= <!--T:19--> | ||
| Line 65: | Line 68: | ||
=Node types and characteristics= <!--T:6--> | =Node types and characteristics= <!--T:6--> | ||
Cedar has a total of 58,416 CPU cores for computation, and 584 GPU devices. | |||
Cedar has a total of 58,416 CPU cores for computation, and 584 GPU devices | |||
<!--T:7--> | <!--T:7--> | ||
| Line 93: | Line 95: | ||
<!--T:10--> | <!--T:10--> | ||
All nodes have local (on-node) temporary storage. GPU nodes have a single 800GB SSD drive. All other compute nodes have two 480GB SSD drives, for a total raw capacity of 960GB. Best practice to access node-local storage is to use the directory generated by [[Running jobs|Slurm]], $SLURM_TMPDIR. | All nodes have local (on-node) temporary storage. GPU nodes have a single 800GB SSD drive. All other compute nodes have two 480GB SSD drives, for a total raw capacity of 960GB. Best practice to access node-local storage is to use the directory generated by [[Running jobs|Slurm]], $SLURM_TMPDIR. | ||
== Choosing a node type == <!--T:27--> | == Choosing a node type == <!--T:27--> | ||
Most applications will run on either Broadwell or Skylake nodes, and performance differences are expected to be small compared to job waiting times. Therefore we recommend that you do not select a specific node type for your jobs. If it is necessary, use <code>--constraint=skylake</code> or <code>--constraint=broadwell</code>. See [[Running_jobs#Specifying_a_CPU_architecture|Specifying a CPU architecture]]. | Most applications will run on either Broadwell or Skylake nodes, and performance differences are expected to be small compared to job waiting times. Therefore we recommend that you do not select a specific node type for your jobs. If it is necessary, use <code>--constraint=skylake</code> or <code>--constraint=broadwell</code>. See [[Running_jobs#Specifying_a_CPU_architecture|Specifying a CPU architecture]]. | ||
== Performance ==<!--T:17--> | |||
Theoretical peak double precision performance of Cedar is 936 teraflops for CPUs, plus 2,744 for GPUs, yielding over 3.6 petaflops of theoretical peak double precision performance. 22 fully connected "islands" of 32 base or large nodes each have 1024 cores in a fully non-blocking topology (Omni-Path fabric), with each island designed to yield over 30 teraflops of double-precision performance (measured with high performance LINPACK). There is a 2:1 blocking factor between the 1024 core islands. | |||
<!--T:16--> | <!--T:16--> | ||