Begin section on parallelization

Author: domfournier

docs/_toc.yml

@@ -16,6 +16,7 @@ chapters:
   - file: fundamentals/joint_inversion
   - file: fundamentals/depth_of_investigation
   - file: fundamentals/optimization
+  - file: fundamentals/parallelization
 - file: tutorials/introduction
   sections:
   - file: tutorials/background

docs/fundamentals/images/distributed_parallelization.svg

@@ -0,0 +1,49 @@
+.. _parallelization:
+
+Parallelization
+===============
+
+For a given inversion routine, the problem can be decomposed into a series of sub-problems, or tiles, each assigned a mesh and a survey. During the inverse process, predicted data and derivatives are continuously requested from the sub-problems. These operations are parallelized within each sub-problem, as well as externally such that sub-problems can be computed concurrently.
+
+
+.. figure:: ./images/distributed_parallelization.svg
+    :align: center
+    :width: 80%
+
+    Schematic representation of the computing elements of a tiled inversion. Each tile is assigned a mesh and a survey, with array operations parallelized by dask bookending a direct solvers. The tiles can be distributed across multiple workers, each with a limited number of threads to optimize performance. Only 1-dimensional arrays are returned to the main process.
+
+This following sections describe the different level of parallelization used by the inversion routines and how to optimize resources.
+
+
+Direct Solvers
+--------------
+
+The direct solvers are used for all methods evaluated by partial differential (PDE) equations, such as electromagnetics and electric methods. The `Pardiso <https://github.com/simpeg/pydiso>`_ and `Mumps <https://gitlab.kwant-project.org/kwant/python-mumps>`_ solvers are parallelized using OpenMP. Note that the current implementation of the solvers are not thread-safe, and can therefore not be shared within parallel processes.
+
+The number of threads used by the solvers can be set by running the command
+
+.. code-block::
+
+    set OMP_NUM_THREADS=X
+
+before launching the python program. Alternatively, setting ``OMP_NUM_THREADS`` as a local environment variable will set it permanently. The default value is the number of threads available on the machine.
+
+Dask
+----
+
+Most operations related to generating arrays are handled by the `dask <https://www.dask.org/>`_ library. A mixture of dask.arrays and dask.delayed calls are used to parallelize the computations across multiple threads. If a direct solver is involved, the dask operations are bookending the solver to avoid thread-safety issues. Otherwise, the dask operations are performed in parallel across the available threads.
+
+
+Dask.distributed
+----------------
+
+For large systems, such as High-Performance Computing (HPC) clusters, the ``dask.distributed`` library can be used to distribute the computation from tiles across multiple ``workers``. It has been found that the performance of direct solvers tend to saturate on large numbers of threads. By spawning multiple processes, each with a limited number of threads, the performance can be improved by running multiple tiles in parallel. The number of workers and threads per worker can be set with the following parameters added to the ui.json file:
+
+.. code-block::
+
+    {
+        ...
+        "n_workers": X,
+        "n_threads": Y,
+        "performance_report": true
+    }

docs/fundamentals/parallelization.rst

@@ -1,4 +1,4 @@
-.. _plate_simulation_index:
+.. _plate_simulation:
 
 Plate Simulation
 ================

docs/plate-simulation/images/sweep/landing.png

@@ -3,8 +3,15 @@
 Batch Simulations
 =================
 
-The Plate Sweep module provides a user interface for generating and running a batch of simulations by sweeping one or more of the input parameters.  The user can select which parameters to sweep and the range of values for each parameter.  The results of each simulation are stored in a ``*.geoh5`` file named with a unique identifier.
+The Plate Sweep module provides a user interface for generating and running a batch of simulations by sweeping one or more of the input parameters.  The user can select which parameters to sweep and the range of values for each parameter.
 
+.. figure:: /plate-simulation/images/sweep/landing.png
+    :align: center
+    :width: 80%
+
+The results of each simulation are stored in a ``*.geoh5`` file named with a unique identifier. A ``summary.xls`` file can be generated to track the parameters used in previous sweeps.
+
+The following sections describe the user interface, inputs, and methodology of the Plate Sweep module.
 
 
 Interface
@@ -22,15 +29,16 @@ Inputs
 - **Plate simulation**: A Plate Simulation group that contains the input parameters for a single plate simulation, as well as the connection to a SimPEG Forward group. Parameters that are not included in the sweep will be taken from this group and used for all simulations.
 - **Output directory**: A directory where the results of each simulation will be stored. Each simulation will be saved in a separate ``*.geoh5`` file named with a unique identifier. The directory is created if it does not exist, otherwise simulations are appended to it.
 - **Generate summary file**: A boolean option to generate a summary file in the output directory. The summary file is a ``*.xls`` file that contains the input parameters and results of each simulation, allowing users to easily sort over the range of simulation parameters.
-- **Sweep block**: For each the following parameters, users can choose a **starting**, **ending** and **step** value to sweep over a range of values. The application will generate a simulation for each value in the range, while keeping all other parameters constant.
-    - **Background**: Over-writing the
+- **[Sweep block]**: For every parameter of `Plate Simulation <plate_simulation>`_, users can choose a **starting**, **ending** and **step** value to sweep over a range of values. The application will generate a simulation for each value in the range, while keeping all other parameters constant. If a parameter is not included in the sweep, the value set in the input Plate Simulation group will be used for all simulations.
+
 
 Methodology
 -----------
 
-Something
+This section provides a brief overview of the methodology used in the Plate Sweep module. For more details on the underlying algorithms and implementation, please refer to the source code.
+
+After loading the input parameters from the Plate Simulation group, the application generates a list of parameter combinations based on the specified sweep ranges. For each combination of parameters, a unique identifier is generated using a hash system. If this unique identifier already exists in the ``Output director``, the simulation is skipped. Otherwise a copy of the original input ``geoh5`` is created. The target file is then opened and the Plate Simulation group is modified with the corresponding parameters before running the simulation. The results are saved in the target file, and the process is repeated for the next combination of parameters until all combinations have been processed.
 
+If the dask.distributed library is enabled, the simulations are run in parallel using a local cluster. Otherwise, the simulations are run sequentially.
 
-Tutorial
---------
-To be added.
+Finally, if the option to generate a summary file is enabled, a routine extracts parameters from all ``*.geoh5`` files present in the ``Output director`` and tabulates them in a ``summary.xls`` file for easy reference and analysis.