Merge pull request #1536 from Libensemble/docs/HPC_running

Docs/hpc running

Author: shuds13

docs/data_structures/data_structures.rst

@@ -8,9 +8,9 @@ See :ref:`here<libe-module>` for instruction on constructing a complete workflow
    :maxdepth: 2
    :caption: libEnsemble Specifications:
 
-   sim_specs
-   gen_specs
    libE_specs
+   gen_specs
+   sim_specs
    alloc_specs
    platform_specs
    persis_info

docs/data_structures/libE_specs.rst

@@ -1,7 +1,7 @@
 .. _datastruct-libe-specs:
 
-General Specs
-=============
+LibE Specs
+==========
 
 libEnsemble is primarily customized by setting options within a ``LibeSpecs`` class or dictionary.
 
@@ -10,11 +10,10 @@ libEnsemble is primarily customized by setting options within a ``LibeSpecs`` cl
     from libensemble.specs import LibeSpecs
 
     specs = LibeSpecs(
-        comm=MPI.COMM_WORLD,
-        comms="mpi",
-        save_every_k_gens=1000,
+        gen_on_manager=True,
+        save_every_k_gens=100,
         sim_dirs_make=True,
-        ensemble_dir_path="/scratch/ensemble",
+        nworkers=4
     )
 
 .. dropdown:: Settings by Category
@@ -272,8 +271,8 @@ libEnsemble is primarily customized by setting options within a ``LibeSpecs`` cl
                     ``disable_resource_manager`` is set.
 
                 **dedicated_mode** [bool] = ``False``:
-                    Disallow any resources running libEnsemble processes (manager and workers)
-                    from being valid targets for app submissions.
+                    Instructs libEnsemble’s MPI executor not to run applications on nodes where
+                    libEnsemble processes (manager and workers) are running.
 
                 **zero_resource_workers** [list of ints]:
                     List of workers (by IDs) that require no resources. For when a fixed mapping of workers

docs/images/centralized_new_detailed.png docs/images/centralized_dedicated.pngdocs/images/centralized_new_detailed.png renamed to docs/images/centralized_dedicated.png

@@ -2,86 +2,105 @@ Example Scheduler Submission Scripts
 ====================================
 
 Below are example submission scripts used to configure and launch libEnsemble
-on a variety of high-powered systems. See :ref:`here<platform-index>` for more
-information about the respective systems and configuration.
+on a variety of high-powered systems. See :ref:`Running on HPC Systems<platform-index>`
+for more information about the respective systems and configuration.
+
+.. note::
+    It is **highly recommended** that the directive lines (e.g., #SBATCH) in batch
+    submission scripts do **NOT** specify  processor, task, or GPU configuration info
+    --- these lines should only specify the number of nodes required.
+
+    For example, do not specify ``#SBATCH --gpus-per-node=4`` in order to use four
+    GPUs on the node, when each worker may use less than this, as this may assign
+    all of the GPUs to a single MPI invocation. Instead,  the configuration should
+    be supplied either
+    :doc:`in the simulation function<../examples/sim_funcs/forces_simf_gpu>`
+    or, if using dynamic resources,
+    :doc:`in the generator<../examples/sim_funcs/forces_simf_gpu_vary_resources>`.
+
 
 General examples
 ----------------
 
-.. dropdown:: Slurm - Basic
+Slurm - Basic
+^^^^^^^^^^^^^
 
-    ..  literalinclude:: ../../examples/libE_submission_scripts/submit_slurm_simple.sh
-        :caption: /examples/libE_submission_scripts/submit_slurm_simple.sh
-        :language: bash
+..  literalinclude:: ../../examples/libE_submission_scripts/submit_slurm_simple.sh
+    :caption: /examples/libE_submission_scripts/submit_slurm_simple.sh
+    :language: bash
 
-.. dropdown:: PBS - Basic
+PBS - Basic
+^^^^^^^^^^^
 
-    ..  literalinclude:: ../../examples/libE_submission_scripts/submit_pbs_simple.sh
-        :caption: /examples/libE_submission_scripts/submit_pbs_simple.sh
-        :language: bash
+..  literalinclude:: ../../examples/libE_submission_scripts/submit_pbs_simple.sh
+    :caption: /examples/libE_submission_scripts/submit_pbs_simple.sh
+    :language: bash
 
-.. dropdown:: LSF - Basic
+LSF - Basic
+^^^^^^^^^^^
 
-    ..  literalinclude:: ../../examples/libE_submission_scripts/submit_lsf_simple.sh
-        :caption: /examples/libE_submission_scripts/submit_lsf_simple.sh
-        :language: bash
+..  literalinclude:: ../../examples/libE_submission_scripts/submit_lsf_simple.sh
+    :caption: /examples/libE_submission_scripts/submit_lsf_simple.sh
+    :language: bash
 
 
 System Examples
 ---------------
 
-.. dropdown:: Aurora
-
-    ..  literalinclude:: ../../examples/libE_submission_scripts/submit_pbs_aurora.sh
-        :caption: /examples/libE_submission_scripts/submit_pbs_aurora.sh
-        :language: bash
-
-.. dropdown:: Frontier (Large WarpX Ensemble)
-
-    ..  literalinclude:: ../../examples/libE_submission_scripts/submit_frontier_large.sh
-        :caption: /examples/libE_submission_scripts/submit_frontier_large.sh
-        :language: bash
+Aurora
+^^^^^^
 
+..  literalinclude:: ../../examples/libE_submission_scripts/submit_pbs_aurora.sh
+    :caption: /examples/libE_submission_scripts/submit_pbs_aurora.sh
+    :language: bash
 
-.. dropdown:: Perlmutter
+Frontier (Large WarpX Ensemble)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-    ..  literalinclude:: ../../examples/libE_submission_scripts/submit_perlmutter.sh
-        :caption: /examples/libE_submission_scripts/submit_perlmutter.sh
-        :language: bash
+..  literalinclude:: ../../examples/libE_submission_scripts/submit_frontier_large.sh
+    :caption: /examples/libE_submission_scripts/submit_frontier_large.sh
+    :language: bash
 
-.. dropdown:: Polaris
+Perlmutter
+^^^^^^^^^^
 
-    ..  literalinclude:: ../../examples/libE_submission_scripts/submit_pbs_polaris.sh
-        :caption: /examples/libE_submission_scripts/submit_pbs_polaris.sh
-        :language: bash
+..  literalinclude:: ../../examples/libE_submission_scripts/submit_perlmutter.sh
+    :caption: /examples/libE_submission_scripts/submit_perlmutter.sh
+    :language: bash
 
-.. dropdown:: Bridges - Central Mode
+Polaris
+^^^^^^^
 
-    ..  literalinclude:: ../../examples/libE_submission_scripts/bridges_submit_slurm_central.sh
-        :caption: /examples/libE_submission_scripts/bridges_submit_slurm_central.sh
-        :language: bash
+..  literalinclude:: ../../examples/libE_submission_scripts/submit_pbs_polaris.sh
+    :caption: /examples/libE_submission_scripts/submit_pbs_polaris.sh
+    :language: bash
 
-.. dropdown:: Bebop - Central Mode
+Bebop
+^^^^^
 
-    ..  literalinclude:: ../../examples/libE_submission_scripts/bebop_submit_slurm_central.sh
-        :caption: /examples/libE_submission_scripts/bebop_submit_slurm_central.sh
-        :language: bash
+..  literalinclude:: ../../examples/libE_submission_scripts/bebop_submit_pbs_central.sh
+    :caption: /examples/libE_submission_scripts/bebop_submit_pbs_central.sh
+    :language: bash
 
-.. dropdown:: Bebop - Distributed Mode
+Bridges - MPI / Central Mode
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-    ..  literalinclude:: ../../examples/libE_submission_scripts/bebop_submit_pbs_distrib.sh
-        :caption: /examples/libE_submission_scripts/bebop_submit_pbs_distrib.sh
-        :language: bash
+..  literalinclude:: ../../examples/libE_submission_scripts/bridges_submit_slurm_central.sh
+    :caption: /examples/libE_submission_scripts/bridges_submit_slurm_central.sh
+    :language: bash
 
-.. dropdown:: Summit (Decommissioned) - On Launch Nodes with Multiprocessing
+.. _slurm_mpi_distributed:
 
-    ..  literalinclude:: ../../examples/libE_submission_scripts/summit_submit_mproc.sh
-        :caption: /examples/libE_submission_scripts/summit_submit_mproc.sh
-        :language: bash
+SLURM - MPI / Distributed Mode (co-locate workers & MPI applications)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-.. dropdown:: Cobalt - Intermediate node with Multiprocessing
+..  literalinclude:: ../../examples/libE_submission_scripts/submit_distrib_mpi4py.sh
+    :caption: /examples/libE_submission_scripts/submit_distrib_mpi4py.sh
+    :language: bash
 
-    .. literalinclude:: ../../examples/libE_submission_scripts/cobalt_submit_mproc.sh
-        :caption: /examples/libE_submission_scripts/cobalt_submit_mproc.sh
-        :language: bash
+Summit (Decommissioned) - On Launch Nodes with Multiprocessing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+..  literalinclude:: ../../examples/libE_submission_scripts/summit_submit_mproc.sh
+    :caption: /examples/libE_submission_scripts/summit_submit_mproc.sh
+    :language: bash