Merge branch 'graphnet-team:main' into 26_01_20_review_docstrings

Author: christianlocatelli

docs/source/data_conversion/data_conversion.rst

@@ -265,8 +265,8 @@ In this example, the writer will save the entire set of extractor outputs - a di
 
 
 
-Two writers are implemented in GraphNeT; the :code:`SQLiteWriter` and :code:`ParquetWriter`, each of which output files that are directly used for
-training by :code:`ParquetDataset` and :code:`SQLiteDataset`.
+Three writers are implemented in GraphNeT; the :code:`SQLiteWriter`, :code:`ParquetWriter`, and :code:`LMDBWriter`, each of which output files that are directly used for
+training by :code:`SQLiteDataset`, :code:`ParquetDataset`, and :code:`LMDBDataset` respectively.
 
 
 

docs/source/datasets/datasets.rst

@@ -155,18 +155,19 @@ It looks like so:
     </details>
 
 
-:code:`SQLiteDataset` & :code:`ParquetDataset`
-----------------------------------------------
+:code:`SQLiteDataset`, :code:`ParquetDataset` & :code:`LMDBDataset`
+--------------------------------------------------------------------
 
-The two specific implementations of :code:`Dataset` exists :
+The three specific implementations of :code:`Dataset` exists :
 
 - `ParquetDataset <https://graphnet-team.github.io/graphnet/api/graphnet.data.parquet.parquet_dataset.html>`_ : Constructs :code:`Dataset` from files created by :code:`ParquetWriter`.
 - `SQLiteDataset <https://graphnet-team.github.io/graphnet/api/graphnet.data.sqlite.sqlite_dataset.html>`_ : Constructs :code:`Dataset` from files created by :code:`SQLiteWriter`.
+- `LMDBDataset <https://graphnet-team.github.io/graphnet/api/graphnet.data.dataset.lmdb.lmdb_dataset.html>`_ : Constructs :code:`Dataset` from files created by :code:`LMDBWriter`.
 
 
 To instantiate a :code:`Dataset` from your files, you must specify at least the following:
 
-- :code:`pulsemaps`: These are named fields in your Parquet files, or tables in your SQLite databases, which store one or more pulse series from which you would like to create a dataset. A pulse series represents the detector response, in the form of a series of PMT hits or pulses, in some time window, usually triggered by a single neutrino or atmospheric muon interaction. This is the data that will be served as input to the `Model`.
+- :code:`pulsemaps`: These are named fields in your Parquet files, or tables in your SQLite or LMDB databases, which store one or more pulse series from which you would like to create a dataset. A pulse series represents the detector response, in the form of a series of PMT hits or pulses, in some time window, usually triggered by a single neutrino or atmospheric muon interaction. This is the data that will be served as input to the `Model`.
 -  :code:`truth_table`: The name of a table/array that contains the truth-level information associated with the pulse series, and should contain the truth labels that you would like to reconstruct or classify. Often this table will contain the true physical attributes of the primary particle — such as its true direction, energy, PID, etc. — and is therefore graph- or event-level (as opposed to the pulse series tables, which are node- or hit-level) truth information.
 -  :code:`features`: The names of the columns in your pulse series table(s) that you would like to include for training; they typically constitute the per-node/-hit features such as xyz-position of sensors, charge, and photon arrival times.
 -  :code:`truth`: The columns in your truth table/array that you would like to include in the dataset.
@@ -225,6 +226,32 @@ Or similarly for Parquet files:
 
     graph = dataset[0]  # torch_geometric.data.Data
 
+Or similarly for LMDB files:
+
+.. code-block:: python
+
+    from graphnet.data.dataset.lmdb.lmdb_dataset import LMDBDataset
+    from graphnet.models.detector.prometheus  import  Prometheus
+    from graphnet.models.graphs  import  KNNGraph
+    from graphnet.models.graphs.nodes  import  NodesAsPulses
+  
+    graph_definition = KNNGraph(
+        detector=Prometheus(),
+        node_definition=NodesAsPulses(),
+        nb_nearest_neighbours=8,
+    )
+
+    dataset = LMDBDataset(
+        path="data/examples/lmdb/prometheus/prometheus-events.lmdb",
+        pulsemaps="total",
+        truth_table="mc_truth",
+        features=["sensor_pos_x", "sensor_pos_y", "sensor_pos_z", "t", ...],
+        truth=["injection_energy", "injection_zenith", ...],
+        graph_definiton = graph_definition,
+    )
+
+    graph = dataset[0]  # torch_geometric.data.Data
+
 It's then straightforward to create a :code:`DataLoader` for training, which will take care of batching, shuffling, and such:
 
 .. code-block:: python
@@ -250,10 +277,10 @@ By default, the following fields will be available in a graph built by :code:`Da
 - :code:`graph[truth_label] for truth_label in truth`: For each truth label in the :code:`truth` argument, the corresponding data is stored as a  :code:`[num_rows, 1]` dimensional tensor. E.g., :code:`graph["energy"] = torch.tensor(26, dtype=torch.float)`
 - :code:`graph[feature] for feature in features`: For each feature given in the :code:`features` argument, the corresponding data is stored as a  :code:`[num_rows, 1]` dimensional tensor. E.g., :code:`graph["sensor_x"] = torch.tensor([100, -200, -300, 200], dtype=torch.float)``
 
-:code:`SQLiteDataset` vs. :code:`ParquetDataset`
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+:code:`SQLiteDataset` vs. :code:`ParquetDataset` vs. :code:`LMDBDataset`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Besides working on different file formats, :code:`SQLiteDataset` and :code:`ParquetDataset` have significant differences, 
+Besides working on different file formats, :code:`SQLiteDataset`, :code:`ParquetDataset`, and :code:`LMDBDataset` have significant differences, 
 which may lead you to choose one over the other, depending on the problem at hand.
 
 :SQLiteDataset: SQLite provides fast random access to all events inside it. This makes plotting and subsampling your dataset particularly easy, 
@@ -265,13 +292,20 @@ which may lead you to choose one over the other, depending on the problem at han
                 This means that the subsampling of your dataset needs to happen prior to the conversion to :code:`parquet`, unlike `SQLiteDataset` which allows for subsampling after conversion, due to it's fast random access.
                 Conversion of files to :code:`parquet` is significantly faster than its :code:`SQLite` counterpart.
 
+:LMDBDataset: LMDB databases produced by :code:`LMDBWriter` store events as key-value pairs with configurable serialization methods (pickle, json, msgpack, dill). 
+              :code:`LMDBDataset` supports two modes: reading raw tables and computing data representations in real-time (similar to :code:`SQLiteDataset`), or reading pre-computed data representations directly from the database for faster access. 
+              LMDB provides fast random access similar to SQLite, while also supporting efficient storage of pre-computed graph representations, making it suitable for scenarios where you want to pre-compute and cache data representations. 
+              LMDB takes up roughly half the space of SQLite, and is therefore a good compromise between SQLite and Parquet.
+
 
 .. note::
 
     :code:`ParquetDataset` is scalable to ultra large datasets, but is more difficult to work with and has a higher memory consumption.
 
     :code:`SQLiteDataset` does not scale to very large datasets, but is easy to work with and has minimal memory consumption.
 
+    :code:`LMDBDataset` provides a balance between SQLite and Parquet, offering fast random access and support for pre-computed representations, making it well-suited for scenarios where data representations are computed once and reused multiple times.
+
 
 Choosing a subset of events using `selection`
 ----------------------------------------------
@@ -297,7 +331,7 @@ would produce a :code:`Dataset` with only those five events.
 
 .. note::
 
-    For :code:`SQLiteDatase`, the :code:`selection` argument specifies individual events chosen for the dataset, 
+    For :code:`SQLiteDataset` and :code:`LMDBDataset`, the :code:`selection` argument specifies individual events chosen for the dataset, 
     whereas for :code:`ParquetDataset`, the :code:`selection` argument specifies which batches are used in the dataset.
 
 
@@ -347,12 +381,14 @@ You can combine multiple instances of :code:`Dataset` from GraphNeT into a singl
     from graphnet.data import EnsembleDataset
     from graphnet.data.parquet import ParquetDataset
     from graphnet.data.sqlite import SQLiteDataset
+    from graphnet.data.dataset.lmdb.lmdb_dataset import LMDBDataset
 
     dataset_1 = SQLiteDataset(...)
     dataset_2 = SQLiteDataset(...)
     dataset_3 = ParquetDataset(...)
+    dataset_4 = LMDBDataset(...)
 
-    ensemble_dataset = EnsembleDataset([dataset_1, dataset_2, dataset_3])
+    ensemble_dataset = EnsembleDataset([dataset_1, dataset_2, dataset_3, dataset_4])
 
 You can find a detailed example `here <https://github.com/graphnet-team/graphnet/blob/main/examples/02_data/04_ensemble_dataset.py>`_ .
 

docs/source/installation/install.rst

@@ -1,68 +1,85 @@
 .. include:: ../substitutions.rst
+===========
+Quick Start
+===========
+Here we provide a quick start guide for getting you started with |graphnet|\ GraphNeT.
 
-Installation
-============
+Installing From Source
+======================
 
-|graphnet|\ GraphNeT is available for Python 3.9 to Python 3.11.
+We recommend installing |graphnet|\ GraphNeT in a separate environment, e.g. using a Python virtual environment or Anaconda (see details on installation `here <https://www.anaconda.com/products/individual>`_).
+With conda installed, you can create a fresh environment like so
 
-.. note::
-   We recommend installing |graphnet|\ GraphNeT in a separate environment, e.g. using a Python virtual environment or Anaconda (see details on installation `here <https://www.anaconda.com/products/individual>`_).
-   With conda installed, you can create a fresh environment like so
+.. code-block:: bash
 
-   .. code-block:: bash
+   # Create the environment with minimal packages
+   conda create --name graphnet_env --no-default-packages python=3.10
+   conda activate graphnet_env
 
-      # Create the environment with minimal packages
-      conda create --name graphnet_env --no-default-packages python=3.10
-      conda activate graphnet_env
+   # Update central packaging libraries
+   pip install --upgrade setuptools packaging
 
-      # Update central packaging libraries
-      pip install --upgrade setuptools packaging
-   
-      # Verify that only wheel, packaging and setuptools are installed
-      pip list 
+   # Verify that only wheel, packaging and setuptools are installed
+   pip list 
 
-      # Now you're ready to proceed with the installation
-Quick Start
------------
+   # Now you're ready to proceed with the installation
+   
 
 .. raw:: html
    :file: quick-start.html
 
 
 When installation is completed, you should be able to run `the examples <https://github.com/graphnet-team/graphnet/tree/main/examples>`_.
 
-Installation in CVMFS (IceCube)
--------------------------------
+Installation into experiment-specific Environments
+--------------------------------------------------
+Users may want to install |graphnet|\ GraphNeT into an environment that is specific to their experiment. This is useful for converting data from the experiment into a deep learning friendly file format, or when deploying models as part of an experiment-specific processing chain.
 
-You may want |graphnet|\ GraphNeT to be able to interface with IceTray, e.g., when converting I3 files to a deep learning friendly file format, or when deploying models as part of an IceTray chain. In these cases, you need to install |graphnet|\ GraphNeT in a Python runtime that has IceTray installed.
+Below are some examples of how to install |graphnet|\ GraphNeT into experiment-specific environments. If your experiment is missing, please feel free to open an issue on the `GitHub repository <https://github.com/graphnet-team/graphnet/issues>`_ and/or contribute a pull request.
 
-To achieve this, we recommend installing |graphnet|\ GraphNeT into a CVMFS with IceTray installed, like so:
+IceTray (IceCube & P-ONE)
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+While |graphnet|\ GraphNeT can be installed into existing IceTray environments that is either built from source or distributed through CVMFS, we highly recommend to instead use our existing Docker images that contain both IceTray and GraphNeT. These images are created by installing GraphNeT into public Docker images from the IceCube Collaboration. 
+
+Details on how to run these images as Apptainer environments are provided in the `Docker & Apptainer Images`_ section.
+
+For users who prefer to install |graphnet|\ GraphNeT directly into a CVMFS environment rather than using Docker/Apptainer images, you can follow the steps below. This example uses PyTorch 2.7.0 (CPU) — adjust the PyTorch version and extras according to the compatibility matrix above.
 
 .. code-block:: bash
-   
+
    # Download GraphNeT
    git clone https://github.com/graphnet-team/graphnet.git
    cd graphnet
+
    # Open your favorite CVMFS distribution
    eval `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh`
    /cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/RHEL_7_x86_64/metaprojects/icetray/v1.5.1/env-shell.sh
-   # Update central utils
-   pip install --upgrade 'pip>=20'
-   pip install wheel setuptools==59.5.0
-   # Install graphnet into the CVMFS as a user
-   pip install --user -r requirements/torch_cpu.txt -e .[torch,develop]
 
+   # Upgrade central packaging libraries
+   pip install --user --upgrade setuptools versioneer
 
-Once installed, |graphnet|\ GraphNeT is available whenever you open the CVMFS locally.
+   # Install PyTorch (CPU)
+   pip3 install --user torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
 
-Installation with km3io (KM3NeT)
------------------------------------------------
+   # Install GraphNeT
+   pip3 install --user -e .[torch-27,develop] -f https://data.pyg.org/whl/torch-2.7.0+cpu.html
 
-This installation is only necessary if you want to process KM3NeT/ARCA or KM3NeT/ORCA files. Processing means converting them from a `.root` offline format into a suitable format for training using |graphnet|. If you already have your KM3NeT data in `SQLite` or `parquet` format and only want to train a model or perform inference on this database, this specific installation is not needed.
+To use |graphnet|\ GraphNeT in a new terminal session, re-activate the CVMFS distribution and the virtual environment:
 
-Note that this installation will add `km3io` ensuring it is built with a compatible versions. The steps below are provided for a conda environment, with an enviroment created in the same way it is done above in this page, but feel free to choose a different enviroment setup.
+.. code-block:: bash
 
-As mentioned, it is highly reommended to create a conda enviroment where your installation is done to do not mess up any dependecy. It can be done with the following commands:
+   eval `/cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh`
+   /cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/RHEL_7_x86_64/metaprojects/icetray/v1.5.1/env-shell.sh
+   source ~/graphnet_venv/bin/activate
+   python -c "import graphnet; print(graphnet.__version__)"
+
+which should print the version of |graphnet|\ GraphNeT.
+
+km3io (KM3NeT)
+~~~~~~~~~~~~~~~~
+Note that this installation will add `km3io` ensuring it is built with a compatible version. The steps below are provided for a conda environment, with an environment created in the same way it is done above in this page, but feel free to choose a different environment setup.
+
+As mentioned, it is highly recommended to create a conda environment where your installation is done to do not mess up any dependency. It can be done with the following commands:
 
 .. code-block:: bash
 
@@ -71,23 +88,23 @@ As mentioned, it is highly reommended to create a conda enviroment where your in
    # Activate the environment and move to the graphnet repository you just cloned. If using conda:
    conda activate <full-path-to-env>
 
-The isntallation of GraphNeT is then done by:
+The installation of GraphNeT is then done by:
 
 .. code-block:: bash
 
    git clone https://github.com/graphnet-team/graphnet.git
    cd graphnet
 
-Choose the appropriate requirements file based on your system. Here there is just an example of installation with PyTorch-2.5.1 but check the matrix above for a full idea of all the versions can be installed.
+Choose the appropriate requirements file based on your system. Here there is just an example of installation with PyTorch-2.5.1 but check the matrix above for a full idea of all the versions that can be installed.
 
-For CPU-only enviroments:
+For CPU-only environments:
 
 .. code-block:: bash
 
    pip3 install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cpu
    pip3 install -e .[torch-25] -f https://data.pyg.org/whl/torch-2.5.1+cpu.html
 
-For GPU enviroments with, for instance, CUDA 11.8 drivers:
+For GPU environments with, for instance, CUDA 11.8 drivers:
 
 .. code-block:: bash
 
@@ -102,5 +119,36 @@ Downgrade setuptools for compatibility between km3io and GraphNeT.
    pip3 install km3io==1.2.0
    
 
-.. note::
-   We recommend installing |graphnet|\ GraphNeT without GPU in clean metaprojects.
+Docker & Apptainer Images
+=========================
+
+We provide Docker images for |graphnet|\ GraphNeT. The list of available Docker images with standalone installations of GraphNeT can be found in DockerHub at https://hub.docker.com/r/rorsoe/graphnet/tags.
+
+New images are created automatically when a new release is published, and when a new PR is merged to the main branch (latest). Each image comes in both GPU and CPU versions, but with a limited selection of pytorch versions. The Dockerfile for the standalone images is `here <https://github.com/graphnet-team/graphnet/blob/main/docker/standalone/Dockerfile>`_.
+
+In compliment to standalone images, we also provide experiment-specific images for:
+
+- `IceCube & P-ONE (IceTray+GraphNeT) <https://hub.docker.com/r/rorsoe/graphnet_icetray/tags>`_ which is built using this `Dockerfile <https://github.com/graphnet-team/graphnet/blob/main/docker/icetray/Dockerfile>`_.
+- KM3NeT (km3io+GraphNeT) (Coming Soon)
+
+
+
+Running Docker images as Apptainer environments
+-----------------------------------------------
+While Docker images require sudo-rights to run, they may be converted to Apptainer images and used as virtual environments - providing a convienient way to run |graphnet|\ GraphNeT without sudo-rights or the need to install it on your system.
+
+To run one of the Docker images as a Apptainer environment, you can use the following command:
+
+.. code-block:: bash
+
+   apptainer exec --cleanenv --env PYTHONNOUSERSITE=1 --env PYTHONPATH= docker://<path_to_image> bash
+
+where <path_to_image> is the path to the image you want to use from the DockerHub. For example, if `rorsoe/graphnet:graphnet-1.8.0-cu126-torch26-ubuntu-22.04` is chosen, an image with GraphNeT 1.8.0 + PyTorch 2.6.0 + CUDA 12.6 installed will open. The additional arguments `--cleanenv --env PYTHONNOUSERSITE=1 --env PYTHONPATH=` ensure that the environment is not contaminated with any other packages that may be installed on your system.
+
+To run one of the images with IceTray+GraphNeT as a Apptainer environment, you can for example use the following command:
+
+.. code-block:: bash
+
+   apptainer exec --cleanenv --env PYTHONNOUSERSITE=1 --env PYTHONPATH= docker://rorsoe/graphnet_icetray:graphnet-1.8.0-cpu-torch26-icecube-icetray-icetray-devel-v1.13.0-ubuntu22.04-2025-02-12 bash
+
+which opens an image with a CPU-installation of GraphNeT 1.8.0 + PyTorch v2.6.0 + IceTray v1.13.0 installed and ready to use. You can replace the image path with the one you want to use from the DockerHub.