Graham: Difference between revisions

[[Transferring_data|Transferring data]]

<!--T:40-->

* A user cannot have more than 1000 jobs, running and queued, at any given moment.  An array job is counted as the number of tasks in the array.

<!--T:24-->

|}

<!--T:21-->

[https://docs.computecanada.ca/mediawiki/images/b/b3/Gp3-network-topo.png Graham high performance interconnect diagram]

<!--T:45-->

Graham has dedicated visualization nodes available at <b>gra-vdi.alliancecan.ca</b> that allow only VNC connections. For instructions on how to use them, see the [[VNC]] page.

A total of 41,548 cores and 520 GPU devices, spread across 1,185 nodes of different types; note that Turbo Boost is activated for the ensemble of Graham nodes.

Note that the amount of available memory is less than the "round number" suggested by hardware configuration.  For instance, "base" nodes do have 128 GiB of RAM, but some of it is permanently occupied by the kernel and OS.  To avoid wasting time by swapping/paging, the scheduler will never allocate jobs whose memory requirements exceed the specified amount of "available" memory.  Please also note that the memory allocated to the job must be sufficient for IO buffering performed by the kernel and filesystem - this means that an IO-intensive job will often benefit from requesting somewhat more memory than the aggregate size of processes.

Graham contains Tesla GPUs from three different generations, listed here in order of age, from oldest to newest.

* P100 Pascal GPUs

P100 is NVIDIA's all-purpose high performance card.  V100 is its successor, with about double the performance for standard computation, and about 8X performance for deep learning computations which can utilize its tensor core computation units.  T4 Turing is the latest card targeted specifically at deep learning workloads - it does not support efficient double precision computations, but it has good performance for single precision, and it also has tensor cores, plus support for reduced precision integer calculations.

<!--T:59-->

These are Graham's default GPU cards.  Job submission for these cards is described on page: [[Using GPUs with Slurm]].  When a job simply request a GPU with --gres=gpu:1 or --gres=gpu:2, it will be assigned to any type of available GPU. If you require a specific type of GPU, please request it. As all Pascal nodes have only 2 P100 GPUs, configuring jobs using these cards is relatively simple.

Graham has a total of 9 Volta nodes.

In 7 of these, four GPUs are connected to each CPU socket (except for one node, which is only populated with 6 GPUs, three per socket).  The other 2 have high bandwidth NVLINK interconnect.

<!--T:65-->

<!--T:53-->

The Volta nodes have a fast local disk, which should be used for jobs if the amount of I/O performed by your job is significant. Inside the job, the location of the temporary directory on fast local disk  is specified by the environment variable $SLURM_TMPDIR. You can copy your input files there at the start of your job script before you run your program and your output files out at the end of your job script. All the files in $SLURM_TMPDIR will be removed once the job ends, so you do not have to clean up that directory yourself.  You can even create Python virtual environments in this temporary space for greater efficiency.  Please see the [[Python#Creating_virtual_environments_inside_of_your_jobs|information on how to do this]].

<!--T:61-->

In this example, two T4 cards per node are requested.

<!--T:67-->