Exaflow

Prerequisites

1. Install python3.10

2. Install poetry It is important to install poetry in isolation, so follow the recommended installation method.

3. Install poetry-shell-plugin

Setup

Environment Setup

1. Install dependencies

   poetry install
   

2. Activate virtual environment

   poetry shell
   

3. Optional To install tab completion for invoke run (replacing bash with your shell)

   source <(poetry run inv --print-completion-script bash)
   

4. Optional pre-commit is included in development dependencies. To install hooks

   pre-commit install
   

Local Deployment

1. Create a deployment configuration file .deployment.toml from the sample file:

   cp .deployment.sample.toml .deployment.toml
   

2. Create the config files that the worker services will use

   inv create-configs
   

3. Install dependencies, start the containers and then the services with

   inv deploy
   

4. Attach to some service's stdout/stderr with

   inv attach --controller
   

   or

   inv attach --worker <WORKER-NAME>
   

5. Restart a specific worker service with

   inv start-worker --localworker1
   

Execute an algorithm

• Examples

  ./run_algorithm -a pca -y leftamygdala lefthippocampus -d ppmi0 -m dementia:0.1
  

  ./run_algorithm -a pearson_correlation -y leftamygdala lefthippocampus -d ppmi0 -m dementia:0.1 -p alpha 0.95
  

Algorithm Development

For a new Exareme3 federated algorithm, start with documentation/new-algorithm-setup.md. It contains the scaffold command, validation gate, required files, and a prompt you can give Codex for end-to-end integration.

Fast path from the repository root:

poetry run python .agents/skills/exaflow-algorithm-scaffold/scripts/integrate_new_algorithm.py --repo-root . --algorithm <algorithm_id> --family <family>

After implementation edits:

poetry run python .agents/skills/exaflow-algorithm-scaffold/scripts/integrate_new_algorithm.py --repo-root . --algorithm <algorithm_id> --family <family> --skip-scaffold

Acknowledgement

This project/research received funding from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Framework Partnership Agreement No. 650003 (HBP FPA).