Deployed 2c3f32c with MkDocs version: 1.6.1

Author: klust

2025-LUST/1_Intro/1_04_LUMI_software_stack/index.html

@@ -4670,8 +4670,7 @@ <h2 id="crayenv-and-lumi-modules">CrayEnv and LUMI modules<a class="headerlink"
 software is available to all LUMI users. There might be license and other restrictions.</p>
 <hr />
 <h2 id="the-partition-module">The <code>partition</code> module<a class="headerlink" href="#the-partition-module" title="Permanent link">&para;</a></h2>
-<p>The <code>LUMI</code> module currently supports five partition modules, but that number may
-be reduced in the future:</p>
+<p>The <code>LUMI</code> module currently supports four partition modules:</p>
 <table>
 <thead>
 <tr>
@@ -4704,6 +4703,28 @@ <h2 id="the-partition-module">The <code>partition</code> module<a class="headerl
 </tbody>
 </table>
 <p>All <code>partition</code> modules also load `craype-network-ofi``.</p>
+<p>However, there are four more hidden partition modules that only play a role in configuring
+EasyBuild and shouldn't be used when running software:</p>
+<ul>
+<li>
+<p><code>partition/common</code> to install some software available in all four above partitions.</p>
+<p>Currently only used for some tools compiled with the system compiler, but it could in fact
+be used to store libraries that are not performance-critical and don't need GPUs (so simply
+compiled for zen2).</p>
+</li>
+<li>
+<p><code>partition/container</code> to install modules that wrap containers, as that software usually does
+    not depend on a version of the CPE anyway. Those container modules are then available in the <code>CrayEnv</code>
+    stack and all of the <code>LUMI</code> stacks.</p>
+</li>
+<li>
+<p><code>partition/system</code> to install some software that is available on all of the system, even just
+    after login. That software should only depend on things in the OS!</p>
+</li>
+<li>
+<p><code>partition/CrayEnv</code> to install packages that will be available in CrayEnv.</p>
+</li>
+</ul>
 <hr />
 <p><em><a href="../1_05_terminology/">[next: Terminology]</a></em></p>
 
@@ -4726,7 +4747,7 @@ <h2 id="the-partition-module">The <code>partition</code> module<a class="headerl
     <span class="md-icon" title="Last update">
       <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M21 13.1c-.1 0-.3.1-.4.2l-1 1 2.1 2.1 1-1c.2-.2.2-.6 0-.8l-1.3-1.3c-.1-.1-.2-.2-.4-.2m-1.9 1.8-6.1 6V23h2.1l6.1-6.1zM12.5 7v5.2l4 2.4-1 1L11 13V7zM11 21.9c-5.1-.5-9-4.8-9-9.9C2 6.5 6.5 2 12 2c5.3 0 9.6 4.1 10 9.3-.3-.1-.6-.2-1-.2s-.7.1-1 .2C19.6 7.2 16.2 4 12 4c-4.4 0-8 3.6-8 8 0 4.1 3.1 7.5 7.1 7.9l-.1.2z"/></svg>
     </span>
-    <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="April 30, 2025 15:24:47">April 30, 2025</span>
+    <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="May 23, 2025 15:04:36">May 23, 2025</span>
   </span>
 
 

2025-LUST/2_Using/2_03_external_modules/index.html

@@ -4703,7 +4703,7 @@ <h2 id="easybuild-metadata-for-external-modules">EasyBuild Metadata for external
 and key-value assignments specific to that module.</p>
 <p>The external modules metadata file can be also defined with the corresponding environment variable:
 <div class="highlight"><pre><span></span><code>echo $EASYBUILD_EXTERNAL_MODULES_METADATA 
-/apps/common/UES/jenkins/production/easybuild/cpe_external_modules_metadata-21.04.cfg
+/appl/lumi/LUMI-SoftwareStack/easybuild/config/external_modules_metadata-CPE-24.03.cfg
 </code></pre></div></p>
 <p>The following keys are 
 <a href="https://docs.easybuild.io/en/latest/Using_external_modules.html#supported-metadata-values">supported by EasyBuild</a>:</p>
@@ -4748,7 +4748,7 @@ <h2 id="easybuild-metadata-for-external-modules">EasyBuild Metadata for external
     <span class="md-icon" title="Last update">
       <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M21 13.1c-.1 0-.3.1-.4.2l-1 1 2.1 2.1 1-1c.2-.2.2-.6 0-.8l-1.3-1.3c-.1-.1-.2-.2-.4-.2m-1.9 1.8-6.1 6V23h2.1l6.1-6.1zM12.5 7v5.2l4 2.4-1 1L11 13V7zM11 21.9c-5.1-.5-9-4.8-9-9.9C2 6.5 6.5 2 12 2c5.3 0 9.6 4.1 10 9.3-.3-.1-.6-.2-1-.2s-.7.1-1 .2C19.6 7.2 16.2 4 12 4c-4.4 0-8 3.6-8 8 0 4.1 3.1 7.5 7.1 7.9l-.1.2z"/></svg>
     </span>
-    <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="May 6, 2025 15:20:25">May 6, 2025</span>
+    <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="May 23, 2025 15:04:36">May 23, 2025</span>
   </span>
 
 

2025-LUST/4_EasyBuild_on_LUMI/4_01_managing_software_on_LUMI/index.html

@@ -1055,6 +1055,15 @@
       </ul>
     </nav>
 
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#managing-versions-of-dependencies" class="md-nav__link">
+    <span class="md-ellipsis">
+      Managing versions of dependencies
+    </span>
+  </a>
+  
 </li>
 
     </ul>
@@ -4719,6 +4728,15 @@
       </ul>
     </nav>
 
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#managing-versions-of-dependencies" class="md-nav__link">
+    <span class="md-ellipsis">
+      Managing versions of dependencies
+    </span>
+  </a>
+  
 </li>
 
     </ul>
@@ -4739,7 +4757,28 @@
 
 <h1 id="managing-easybuild-installed-software-on-lumi">Managing EasyBuild-installed software on LUMI<a class="headerlink" href="#managing-easybuild-installed-software-on-lumi" title="Permanent link">&para;</a></h1>
 <h2 id="repositories-and-their-function">Repositories and their function<a class="headerlink" href="#repositories-and-their-function" title="Permanent link">&para;</a></h2>
-<p>TODO</p>
+<p>There are three LUST-maintained repositories with EasyConfig files:</p>
+<ul>
+<li>
+<p><a href="https://github.com/Lumi-supercomputer/LUMI-SoftwareStack">LUMI-SoftwareStack</a>: 
+    Centrally installed software, but also the whole Lmod configuration and 
+    special modules, and scripts to manage the software stack</p>
+<p>Documented on <a href="https://lumi-supercomputer.github.io/LUMI-SoftwareStack/">lumi-supercomputer.github.io/LUMI-SoftwareStack</a>,
+automatically generated by a GitHub action</p>
+</li>
+<li>
+<p><a href="https://github.com/Lumi-supercomputer/LUMI-EasyBuild-contrib">LUMI-EasyBuild-contrib</a>: User-installable software</p>
+</li>
+<li>
+<p><a href="https://github.com/Lumi-supercomputer/LUMI-EasyBuild-containers">LUMI-EasyBuild-containers</a>: 
+    Module wrappers for containers: AMD AI containers and CCPE</p>
+</li>
+</ul>
+<p>Users can have their own repo also. When it is cloned as <code>UserRepo</code> in <code>$EBU_USER_PREFIX</code>,
+it will be used by EasyBuildwhen configured with <code>LUMI-EasyBuild-user</code>.</p>
+<p>There is another repository also, 
+<a href="https://github.com/Lumi-supercomputer/LUMI-EasyBuild-docs">LUMI-EasyBuild-docs</a> 
+to manage building the LUMI Software Library</p>
 <h2 id="creating-and-installing-a-new-release-of-the-lumi-software-stack">Creating and installing a new release of the LUMI software stack<a class="headerlink" href="#creating-and-installing-a-new-release-of-the-lumi-software-stack" title="Permanent link">&para;</a></h2>
 <p>TODO not complete</p>
 <ul>
@@ -4808,7 +4847,7 @@ <h2 id="creating-and-installing-a-new-release-of-the-lumi-software-stack">Creati
 </ul>
 </li>
 <li>
-<p>Before creating a new release, add the release name (YYYyMMDD) to the 
+<p>Before creating a new release, add the release name (YYYYMMDD) to the 
     <code>docs/whats_new.md</code> with a description of the changes/additions. This page
     will also appear in the LUMI Software Library.</p>
 </li>
@@ -4822,7 +4861,7 @@ <h3 id="lumi-software-library">LUMI Software Library<a class="headerlink" href="
 The script is very naive when it tries to renumber the title levels. It goes wrong when a
 code block is included that starts on the first column and contains lines starting with <code>#</code>.
 The workaround is to indent such a code block with one space.</p>
-<p>Files stored with the EasyConfig:</p>
+<p>Files stored with the EasyConfigs of a package:</p>
 <ul>
 <li>
 <p><code>USER.md</code>: Optional, extra information for regular users.</p>
@@ -4847,6 +4886,28 @@ <h3 id="lumi-software-library">LUMI Software Library<a class="headerlink" href="
 </ul>
 </li>
 </ul>
+<p>Some interesting examples to look at:</p>
+<ul>
+<li>
+<p><a href="https://lumi-supercomputer.github.io/LUMI-EasyBuild-docs/q/QuantumESPRESSO/">QuantumESPRESSO</a>:
+    uses the full range of tricks. Check the <a href="https://github.com/Lumi-supercomputer/LUMI-EasyBuild-contrib/tree/main/easybuild/easyconfigs/q/QuantumESPRESSO">files in the GitHub</a>:
+    the pictures are in the 'files' subdirectory</p>
+</li>
+<li>
+<p><a href="https://lumi-supercomputer.github.io/LUMI-EasyBuild-docs/e/ELPA/">ELPA</a> used to be in the
+    central stack but was moved out because we needed multiple configurations in a single 
+    toolchain and also had too many issues to get it to work correctly. </p>
+<p>It has a 'README.md' file in both 
+<a href="https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/easybuild/easyconfigs/e/ELPA">LUMI-SoftwareStack</a>
+and <a href="https://github.com/Lumi-supercomputer/LUMI-EasyBuild-contrib/tree/main/easybuild/easyconfigs/e/ELPA">LUMI-EasyBuild-contrib</a>
+but as we are phasing the 22.08 toolchain out, that example may have become irrelevant with a dead
+link by the time you read this.</p>
+</li>
+<li>
+<p><a href="https://lumi-supercomputer.github.io/LUMI-EasyBuild-docs/b/buildtools/">buildtools</a>: Made sure that users can
+    actually find the included packages through the search box by mentioning them on the page.</p>
+</li>
+</ul>
 <h3 id="lumi-softwarestack-repository">LUMI-SoftwareStack repository<a class="headerlink" href="#lumi-softwarestack-repository" title="Permanent link">&para;</a></h3>
 <p>The repository contains a <code>docs</code> subdirectory with markdown-based documentation. </p>
 <p>The <code>docs/config</code> subdirectory contains the <code>mkdocs.yml</code> file for the documentation,
@@ -4860,6 +4921,41 @@ <h3 id="documenting-the-actual-installation-process-on-lumi-uan06">Documenting t
 <a href="https://wiki.eduuni.fi/spaces/CSCLUST/pages/266410121/Change+log+for+the+software+stack+in+appl+lumi">"Change log for the software stack in /appl/lumi" page in the LUST eduuni wiki</a>.</p>
 <p>This is done <em>before</em> actually doing the installation and has been tested as much as possible on a copy of the software stack
 (currently a personal copy, could be a LUST one in 462000008).</p>
+<h2 id="managing-versions-of-dependencies">Managing versions of dependencies<a class="headerlink" href="#managing-versions-of-dependencies" title="Permanent link">&para;</a></h2>
+<p>When building a new software stack, we often want to change the version of packages to a newer version.
+This implies that all EasyConfigs where the package appears as a dependency, should also change.
+The <code>dependencies</code> parameter is way too complex to tackle with <code>try-amend</code> so we needed another 
+approach.</p>
+<p>On LUMI, we declare the version of packages as much as possible through variables, e.g.,
+<code>local_zlib_version</code>. We maintain files with all version variables in 
+<a href="https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/easybuild/easyconfigs">LUMI-SoftwareStack</a>
+(look vor the <code>versions-yy.MM.txt</code> files) and
+<a href="https://github.com/Lumi-supercomputer/LUMI-EasyBuild-contrib/tree/main/easybuild/easyconfigs">LUMI-EasyBuild-contrib</a>
+(look for the <code>versions-contrib-YY.MM.txt</code> files). 
+For Contrib this only makes sense for packages that are likely to be used as a dependency and are typically available
+in only one version (though for, e.g., PLUMED., we simply define separate variables for PLUMED 8.X and 9.X).</p>
+<p>There are two scripts to update toolchains and dependency versions:</p>
+<ul>
+<li>
+<p><a href="https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/blob/main/tools/upgrade-tc.py">upgrade-tc.py</a>
+    is a script developed at CSCS to update the toolchain version in an EasyConfig and also adapt its name
+    accordingly.</p>
+</li>
+<li>
+<p><a href="https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/blob/main/tools/upgrade-locals.lua">upgrade-locals.lua</a>
+    is a script developed by LUST to use the variables in the <code>version-YY.MM.txt</code> files to update the corresponding
+    variables in the EasyConfigs. The only downside is that it cannot adapt the name of the EasyConfig. That still has
+    to be done manually.</p>
+</li>
+</ul>
+<p>It is easiest to use those scripts through aliases. E.g., Kurt Lust uses the following aliases to adapt EasyConfig files
+for LUMI/24.03:</p>
+<div class="highlight"><pre><span></span><code>alias tc2403=&quot;$installroot/LUMI-SoftwareStack/tools/upgrade-tc.py --metadata $installroot/LUMI-SoftwareStack/easybuild/config/external_modules_metadata-CPE-24.03.cfg --version 24.03 --filenames&quot;
+alias locals2403=&quot;$installroot/LUMI-SoftwareStack/tools/upgrade-locals.lua $installroot/LUMI-SoftwareStack/easybuild/easyconfigs/versions-24.03.txt&quot;
+alias contrib2403=&quot;$installroot/LUMI-SoftwareStack/tools/upgrade-locals.lua $installroot/LUMI-EasyBuild-contrib/easybuild/easyconfigs/versions-contrib-24.03.txt&quot;
+</code></pre></div>
+<p>(with <code>installroot</code> an environment variable pointing to the local installation he uses; you could use <code>/appl/lumi</code> for that but would
+probably like to refer to your own copy of <code>LUMI-EasyBuild-contrib</code>).</p>
 <p><em><a href="../4_02_tips_and_tricks/">[Next: EasyBuild tips &amp; tricks]</a></em></p>
 
 
@@ -4881,7 +4977,7 @@ <h3 id="documenting-the-actual-installation-process-on-lumi-uan06">Documenting t
     <span class="md-icon" title="Last update">
       <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M21 13.1c-.1 0-.3.1-.4.2l-1 1 2.1 2.1 1-1c.2-.2.2-.6 0-.8l-1.3-1.3c-.1-.1-.2-.2-.4-.2m-1.9 1.8-6.1 6V23h2.1l6.1-6.1zM12.5 7v5.2l4 2.4-1 1L11 13V7zM11 21.9c-5.1-.5-9-4.8-9-9.9C2 6.5 6.5 2 12 2c5.3 0 9.6 4.1 10 9.3-.3-.1-.6-.2-1-.2s-.7.1-1 .2C19.6 7.2 16.2 4 12 4c-4.4 0-8 3.6-8 8 0 4.1 3.1 7.5 7.1 7.9l-.1.2z"/></svg>
     </span>
-    <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="May 14, 2025 09:41:11">May 14, 2025</span>
+    <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="May 23, 2025 15:04:36">May 23, 2025</span>
   </span>
 
 

2025-LUST/4_EasyBuild_on_LUMI/4_02_tips_and_tricks/index.html

@@ -4678,6 +4678,14 @@ <h2 id="long-lists-of-elements-for-preconfigopts-or-options-for-configopts">Long
 There is also plenty of space left on each line to explain why an option is used or what it
 means, to make the job easier for others who may want to update or customise this EasyConfig.</p>
 <h2 id="more-to-follow">More to follow....<a class="headerlink" href="#more-to-follow" title="Permanent link">&para;</a></h2>
+<ul>
+<li>
+<p>Static and shared libraries in CMakeMake packages (and using lib instead of lib64)</p>
+</li>
+<li>
+<p>Copying the license information</p>
+</li>
+</ul>
 <p><em><a href="../../5_00_additional_reading/">[Next: Additional reading]</a></em></p>
 
 
@@ -4699,7 +4707,7 @@ <h2 id="more-to-follow">More to follow....<a class="headerlink" href="#more-to-f
     <span class="md-icon" title="Last update">
       <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M21 13.1c-.1 0-.3.1-.4.2l-1 1 2.1 2.1 1-1c.2-.2.2-.6 0-.8l-1.3-1.3c-.1-.1-.2-.2-.4-.2m-1.9 1.8-6.1 6V23h2.1l6.1-6.1zM12.5 7v5.2l4 2.4-1 1L11 13V7zM11 21.9c-5.1-.5-9-4.8-9-9.9C2 6.5 6.5 2 12 2c5.3 0 9.6 4.1 10 9.3-.3-.1-.6-.2-1-.2s-.7.1-1 .2C19.6 7.2 16.2 4 12 4c-4.4 0-8 3.6-8 8 0 4.1 3.1 7.5 7.1 7.9l-.1.2z"/></svg>
     </span>
-    <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="May 14, 2025 09:41:11">May 14, 2025</span>
+    <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="May 23, 2025 15:04:36">May 23, 2025</span>
   </span>