@@ -15,7 +15,34 @@ On this page we focus more on the technical implementation behind it.
1515LUMI has different node types providing compute resources:
1616
1717- LUMI has 16 login nodes, though many of those are reserved for special purposes and not
18- available to all users. TODO
18+ available to all users. These login nodes have a zen2 CPU. These nodes have a SlingShot 10
19+ interconnect.
20+ - There are 1536 regular CPU compute nodes in a partition denoted as LUMI-C. These
21+ compute nodes have a zen3 CPU and run a reduced version of SUSE Linux optimised
22+ by Cray to reduce OS jitter. These nodes will in the future be equipped with a
23+ SlingShot 11 interconnect card.
24+ - There are 2560 GPU compute nodes in a partition denoted as LUMI-G. These nodes have
25+ a single zen3-based CPU with optimised I/O die linked to 4 AMD MI250X GPUs. Each node
26+ has 4 SlingShot 11 interconnect cards, one attached to each GPU.
27+ - The interactive data analytics and visualisation partition is really two different partitions
28+ from the software point-of-view:
29+ - 8 nodes are CPU-only but differ considerably from the regular compute nodes,
30+ not only in the amount of memory. These nodes are equipped with zen2 CPUs
31+ and in that sense comparable to the login nodes. They also have local SSDs
32+ and are equipped with SlingShot 10 interconnect cards (2 each???)
33+ - 8 nodes have zen2 CPUs and 8 NVIDIA A40 GPUs each, and have 2 SlingShot 10
34+ interconnect cards each.
35+ - The early access platform (EAP) has 14 nodes equiped with a single 64-core
36+ zen2 CPU and 4 AMD MI100 GPUS. Each node has a single SlingShot 10 interconnect
37+ and also local SSDs.
38+
39+ SlingShot 10 and SlingShot 11 are different software-wise. SlingShot 10 uses a
40+ Mellanox CX5 NIC that support both OFI and UCX, and hence can also use the
41+ UCX version of Cray MPICH. SlingShot 11 uses a NIC code-named Cassini and
42+ supports only OFI with an OFI provider specific for the Cassini NIC. However,
43+ given that the nodes that are equipped with SlingShot 10 cards are not meant
44+ to be used for big MPI jobs, we build our software stack solely on top of
45+ libfabric and Cray MPICH.
1946
2047
2148---
@@ -46,7 +73,9 @@ On LUMI, two types of software stacks are currently offered:
4673 A deliberate choice was made to only offer a limited number of software
4774 packages in the globally installed stack as the setup of redundancy on LUMI
4875 makes it difficult to update the stack in a way that is guaranteed to not
49- affect running jobs and as a large central stack is also hard to manage.
76+ affect running jobs and as a large central stack is also hard to manage, especially
77+ as we expect frequent updates to the OS and compiler infrastructure in
78+ the first years of operation.
5079 However, the EasyBuild setup is such that users can easily install
5180 additional software in their home or project directory using EasyBuild build
5281 recipes that we provide or they develop, and that software will fully
@@ -72,12 +101,13 @@ set of target modules after loading the ``CrayEnv`` module.
72101The `` LUMI `` module currently supports four partition modules, but that number may
73102be reduced in the future:
74103
75- | Partition | CPU target | Accelerator |
76- | :----------------| -----------------------| :----------------------------|
77- | `` partition/L `` | `` craype-x86-rome `` | `` craype-accel-host `` |
78- | `` partition/C `` | `` craype-x86-milan `` | `` craype-accel-host `` |
79- | `` partition/G `` | `` craype-x86-trento `` | `` craype-accel-amd-gfx90a `` |
80- | `` partition/D `` | `` craype-x86-rome `` | `` craype-accel-nvidia80 `` |
104+ | Partition | CPU target | Accelerator |
105+ | :------------------| -----------------------| :----------------------------|
106+ | `` partition/L `` | `` craype-x86-rome `` | `` craype-accel-host `` |
107+ | `` partition/C `` | `` craype-x86-milan `` | `` craype-accel-host `` |
108+ | `` partition/G `` | `` craype-x86-trento `` | `` craype-accel-amd-gfx90a `` |
109+ | `` partition/D `` | `` craype-x86-rome `` | `` craype-accel-nvidia80 `` |
110+ | `` partition/EAP `` | `` craype-x86-rome `` | `` craype-accel-amd-gfx908 `` |
81111
82112All `` partition `` modules also load `craype-network-ofi``.
83113
0 commit comments