regression 2 renamed regression + documentation updated

Author: janursa

common

@@ -3,7 +3,7 @@ GRN evaluation
 =================
 The evaluation metrics used in geneRNIB are summarized below. For a detailed description of each metric, refer to the geneRNIB paper.
 
-We originally defined **eight evaluation metrics**, grouped into three categories: **Regression 1, Regression 2, and Wasserstein Distance**. 
+We originally defined **eight evaluation metrics**, grouped into three categories: **Regression 1, Regression, and Wasserstein Distance**. 
 However, we recently removed **Regression 1** as it did not prove to be effective for perturbational settings. 
 
 - The **regression-based metrics** assess the predictive power of an inferred GRN by using regression models to predict perturbation data (evaluation data) based on the feature space constructed from the inferred network.  

docs/build/html/_sources/evaluation.rst.txt

@@ -76,7 +76,7 @@
   <section id="grn-evaluation">
 <h1>GRN evaluation<a class="headerlink" href="#grn-evaluation" title="Link to this heading"></a></h1>
 <p>The evaluation metrics used in geneRNIB are summarized below. For a detailed description of each metric, refer to the geneRNIB paper.</p>
-<p>We originally defined <strong>eight evaluation metrics</strong>, grouped into three categories: <strong>Regression 1, Regression 2, and Wasserstein Distance</strong>.
+<p>We originally defined <strong>eight evaluation metrics</strong>, grouped into three categories: <strong>Regression 1, Regression, and Wasserstein Distance</strong>.
 However, we recently removed <strong>Regression 1</strong> as it did not prove to be effective for perturbational settings.</p>
 <ul class="simple">
 <li><p>The <strong>regression-based metrics</strong> assess the predictive power of an inferred GRN by using regression models to predict perturbation data (evaluation data) based on the feature space constructed from the inferred network.</p></li>

docs/build/html/evaluation.html

@@ -1,15 +1,22 @@
 
 GRN evaluation
 =================
-The evaluation metrics used in geneRNIB are summarized below. For a detailed description of each metric, refer to the geneRNIB paper.
-
+The evaluation metrics used in geneRNIB are summarized below. 
 
 
 .. image:: images/metrics.png
    :width: 90%
    :align: center
 ----
 
+.. image:: images/datasets_metrics.png
+   :width: 90%
+   :align: center
+----
+
+
+For a detailed description of each metric, refer to the geneRNIB paper.
+
 The evaluation metrics expect the inferred network to be in the form of an AnnData object with specific format as explained here. 
 It should be noted that the metric currently evaluate only the **top TF-gene pairs**, currently limited to **50,000 edges**, ranked by their assigned weight.  
 
@@ -21,6 +28,7 @@ The inferred network should have a tabular format with the following columns:
 
 See `resources/grn_benchmark/prior/collectri.h5ad` for an example of the expected format.
 
+## Running GRN evaluation using standard pipeline
 
 To run the evalution for a given GRN and dataset, use the following command:
 ```bash
@@ -31,3 +39,16 @@ example command:
 ```bash
 bash scripts/run_grn_evaluation.sh --prediction=resources/grn_models/op/collectri.h5ad --save_dir=output/ --dataset=op --build_images=true 
 ```
+
+
+## Running GRN evaluation without docker
+Considering that Docker is not supported by certtain systems, you can run the evaluation without Docker by following these steps:
+
+```bash
+bash src/metrics/all_metrics/run_local.sh --dataset <dataset_name> --prediction=<inferred GRN (e.g.collectri.h5ad)> --score <output_score_file.h5ad> --num_workers <number_of_workers>
+```
+
+example command:
+```bash
+bash src/metrics/all_metrics/run_local.sh --dataset op --prediction=resources/grn_models/op/collectri.h5ad --score=output_score_file.h5ad --num_workers=20
+```

docs/source/evaluation.rst

@@ -2,40 +2,39 @@ Documentation for Gene Regulatory Network Inference Benchmark (geneRNIB)
 ========================================================================
 
 
-geneRNIB is a living benchmark platform for GRN inference. This platform provides curated datasets for GRN inference and evaluation, standardized evaluation protocols and metrics, computational infrastructure, and a dynamically updated leaderboard to track state-of-the-art methods. It runs novel GRNs in the cloud, offers competition scores, and stores them for future comparisons, reflecting new developments over time.
+geneRNIB is a living benchmark platform for GRN inference. This platform provides curated datasets for GRN inference and evaluation, standardized evaluation protocols and metrics, computational infrastructure, and a dynamically updated leaderboard to track state-of-the-art methods. 
+It runs novel GRNs in the cloud, offers competition scores, and stores them for future comparisons, reflecting new developments over time.
 
-The platform supports the integration of new inference methods, datasets, and protocols. When a new feature is added, previously evaluated GRNs are re-assessed, and the leaderboard is updated accordingly. The aim is to evaluate both the accuracy and completeness of inferred GRNs. It is designed for both single-modality and multi-omics GRN inference.
+The platform supports the integration of new inference methods, datasets, and protocols. When a new feature is added, previously evaluated GRNs are re-assessed, and the leaderboard is updated accordingly. 
+It is designed for both single-modality and multi-omics GRN inference.
 
 .. image:: images/overview.png
    :width: 70%
    :align: center
 ----
 
-This documentation is supplementary to the paper `geneRNIB: a living benchmark for gene regulatory network inference <add a link here>`_ and the `GitHub page <https://github.com/openproblems-bio/task_grn_inference>`_ on the OpenProblems platform. 
+This documentation is supplementary to the paper `geneRNIB: a living benchmark for gene regulatory network inference <https://www.biorxiv.org/content/10.1101/2025.02.25.640181v1.full.pdf>`_ and the `GitHub page <https://github.com/openproblems-bio/task_grn_inference>`_ on the OpenProblems platform. 
 
-To install geneRNIB, see the `GitHub page <https://github.com/openproblems-bio/task_grn_inference>`_.
- 
-For instructions on how to download and access datasets, refer to the :doc:`dataset` section.
+- To install geneRNIB, see the `GitHub page <https://github.com/openproblems-bio/task_grn_inference>`_
+- To download, see :doc:`dataset` page
+- To perform GRN inference using our integrated methods, see :doc:`inference` page
+- To run evaluation metrics, see :doc:`evaluation` page
+- To extend geneRNIB with new methods, metrics, or datasets, see :doc:`extending` page
+- To view the leaderboard of integrated methods, see :doc:`leaderboard` page
 
-For information on evaluation metrics, refer to the :doc:`evaluation` section.
+.. .. image:: images/grn_models.png
+..    :width: 70%
+..    :align: center
+.. ----
 
-To integrate your GRN inference method, metric, or dataset, follow the instructions in the :doc:`extending` section. 
 
-To see the comparitive performance of the integrated GRN inference methods, refer to the :doc:`leaderboard` section.
-
-.. image:: images/grn_models.png
-   :width: 70%
-   :align: center
-----
+.. Pls see the GitHub page for the list of currently integrated methods. The methods are implemented in Python and R, and they can be used to infer GRNs from the datasets provided by geneRNIB.
 
+.. In addition, three baseline methods are integrated into geneRNIB. These methods are used to evaluate the performance of new methods. The baseline methods are:
 
-Pls see the GitHub page for the list of currently integrated methods. The methods are implemented in Python and R, and they can be used to infer GRNs from the datasets provided by geneRNIB.
-
-In addition, three baseline methods are integrated into geneRNIB. These methods are used to evaluate the performance of new methods. The baseline methods are:
-
-- **Negative control**: Randomly assigns weights to edges. GRN inference methods should outperform this method.
-- **Pearson correlation**: Assigns weights based on the Pearson correlation between genes.
-- **Positive control**: Similar to Pearson correlation with the exception that it uses both inference and evaluation dataset to infer the GRN. This method is expected to outperform most methods.
+.. - **Negative control**: Randomly assigns weights to edges. GRN inference methods should outperform this method.
+.. - **Pearson correlation**: Assigns weights based on the Pearson correlation between genes.
+.. - **Positive control**: Similar to Pearson correlation with the exception that it uses both inference and evaluation dataset to infer the GRN. This method is expected to outperform most methods.
 
 
 .. .. list-table:: Authors & contributors
@@ -59,8 +58,8 @@ Contents
 --------
 
 .. toctree::
-
    dataset
+   inference
    evaluation
    extending
    leaderboard

docs/source/images/datasets_metrics.png

@@ -30,12 +30,12 @@ for model in "${models[@]}"; do
 done
 printf '%s\n' "${predictions[@]}"
 
-echo "Running consensus for regression 2"
+echo "Running consensus for Regression"
 datasets=(${DATASET})
 for dataset in "${datasets[@]}"; do
     echo "Running reg2 consensus for dataset: $dataset"
 
-    python src/metrics/regression_2/consensus/script.py \
+    python src/metrics/regression/consensus/script.py \
         --dataset "$dataset" \
         --regulators_consensus "resources/grn_benchmark/prior/regulators_consensus_${dataset}.json" \
         --evaluation_data "resources/grn_benchmark/evaluation_data/${dataset}_bulk.h5ad" \

docs/source/index.rst

@@ -97,19 +97,19 @@ HERE
 
 # --------- COMBINATIONS TO ADD ----------
 
-# append_entry "op" "[regression_1,regression_2, ws_distance]" "[pearson_corr, negative_control, positive_control, 
+# append_entry "op" "[regression_1,regression, ws_distance]" "[pearson_corr, negative_control, positive_control, 
 #                                                                         portia, ppcor, scenic, scprint, grnboost,
 #                                                                         scenicplus, scglue, granie, figr, celloracle]" 
-# append_entry "norman"  "[regression_1,regression_2, ws_distance]" "[pearson_corr, negative_control, positive_control, 
+# append_entry "norman"  "[regression_1,regression, ws_distance]" "[pearson_corr, negative_control, positive_control, 
 #                                                                         portia, ppcor, scenic, scprint, grnboost]"
-# append_entry "adamson"  "[regression_1,regression_2, ws_distance]" "[pearson_corr, negative_control, positive_control, 
+# append_entry "adamson"  "[regression_1,regression, ws_distance]" "[pearson_corr, negative_control, positive_control, 
 #                                                                         portia, ppcor, scenic, grnboost]"
-# append_entry "nakatake"  "[regression_1,regression_2]" "[pearson_corr, negative_control, positive_control, 
+# append_entry "nakatake"  "[regression_1,regression]" "[pearson_corr, negative_control, positive_control, 
 #                                                                         portia, scenic, grnboost]"
-# append_entry "replogle" "[regression_1, regression_2, ws_distance]" "[pearson_corr, negative_control, positive_control, portia, ppcor, scenic, grnboost]"
-# append_entry "replogle" "[regression_1, regression_2, ws_distance]" "[scprint]" "special_case" 
+# append_entry "replogle" "[regression_1, regression, ws_distance]" "[pearson_corr, negative_control, positive_control, portia, ppcor, scenic, grnboost]"
+# append_entry "replogle" "[regression_1, regression, ws_distance]" "[scprint]" "special_case" 
 
-append_entry "xaira_HCT116"  "[regression_1, regression_2]" "[pearson_corr, negative_control, positive_control]"
+append_entry "xaira_HCT116"  "[regression_1, regression]" "[pearson_corr, negative_control, positive_control]"
 
 # --- Final configuration ---
 if [ "$run_local" = true ]; then

scripts/prior/run_consensus.sh

@@ -13,7 +13,7 @@ apply_tf_methods=true
 apply_skeleton=false
 # - specify inputs
 dataset_ids=" op " 
-metric_ids="[regression_1, regression_2, ws_distance]" 
+metric_ids="[regression_1, regression, ws_distance]" 
 method_ids="[pearson_corr,
             negative_control, 
             positive_control, 

scripts/repo/run_benchmark_all.sh

@@ -14,7 +14,7 @@ grn_models_folder="${resources_dir}/grn_models"
 subsample=-2
 max_workers=10
 layer=scgen_pearson
-metric_ids="[regression_1, regression_2]"
+metric_ids="[regression_1, regression]"
 
 param_file="./params/${RUN_ID}.yaml"