Merge branch 'dev' into multiscale_outputs_struct_rework

Author: Samuel-amap

.github/workflows/benchmarks_and_downstream.yml

@@ -2,9 +2,8 @@ name: Benchmarks
 on:
   push:
     branches:
-      - main
-      - Outputs-filtering2
-      - PSE-multiscale-outputs-structure-change
+      - dev
+      - benchmarks-github-action
     tags: "*"
   pull_request:
   workflow_dispatch:
@@ -30,8 +29,8 @@ jobs:
         arch:
           - x64
         package:
-        - {user: PalmStudio, repo: XPalm.jl, branch: PSE-multiscale-outputs-structure-change}
-        - {user: VEZY, repo: PlantBioPhysics.jl, branch: PSE-multiscale-outputs-structure-change}
+       # the group setting is unused atm
+        - {user: VEZY, repo: PlantSimEngine.jl, group: Downstream}
     steps:
       - uses: actions/checkout@v4
       - uses: julia-actions/setup-julia@v2
@@ -51,8 +50,7 @@ jobs:
         run: |
           cd test/downstream
           julia --project --threads 4 --color=yes -e '
-            using Pkg;
-            Pkg.instantiate();
+            using Pkg;            
             include("test-all-benchmarks.jl")'
           rm Manifest.toml
       - name: Store benchmark result

Project.toml

@@ -1,7 +1,7 @@
 name = "PlantSimEngine"
 uuid = "9a576370-710b-4269-adf9-4f603a9c6423"
 authors = ["Rémi Vezy <VEZY@users.noreply.github.com> and contributors"]
-version = "0.11.3"
+version = "0.12.0"
 
 [deps]
 AbstractTrees = "1520ce14-60c1-5f80-bbc7-55ef81b5835c"

docs/make.jl

@@ -22,16 +22,12 @@ makedocs(;
     ), pages=[
         "Home" => "index.md",
         "Introduction" => [
-            #"Organization of the documentation ?"
             "Why PlantSimEngine ?" => "./introduction/why_plantsimengine.md",
             "Why Julia ?" => "./introduction/why_julia.md",
-            #"Overview ?"
-            #"Feature list ? Companion packages ?"
         ],
         "Prerequisites" => [
             "Installing and running PlantSimEngine" => "./prerequisites/installing_plantsimengine.md",
-            "Key Concepts" => "./prerequisites/key_concepts.md", # Key concepts vs terminology ?
-            #"Setup" ?",
+            "Key Concepts" => "./prerequisites/key_concepts.md",
             "Julia language basics" => "./prerequisites/julia_basics.md",
         ],
         "Step by step - Single-scale simulations" => [
@@ -62,7 +58,7 @@ makedocs(;
             "Building a simple plant" => [
                 "A rudimentary plant simulation" => "./multiscale/multiscale_example_1.md",
                 "Expanding the plant simulation" => "./multiscale/multiscale_example_2.md",
-                "Fixing bugs in the plant simulation"=> "./multiscale/multiscale_example_3.md", # TODO illustrate outputs filtering to find the bug
+                "Fixing bugs in the plant simulation"=> "./multiscale/multiscale_example_3.md",
             ],
             "Visualizing our toy plant with PlantGeom"=> "./multiscale/multiscale_example_4.md",
         ], "Troubleshooting and testing" => [
@@ -74,14 +70,11 @@ makedocs(;
             "Public API" => "./API/API_public.md",
             "Example models" => "./API/API_examples.md",
             "Internal API" => "./API/API_private.md",],
-        "Credits" => "credits.md",
         "Improving our documentation" => "documentation_improvement.md",
         "Developer guidelines" => "developers.md",
         "Planned features" => "planned_features.md",
-        #"developer section TODO"
     ]
 )
-# move repeated examples listing to a specific page ?
 
 deploydocs(;
     repo="github.com/VirtualPlantLab/PlantSimEngine.jl.git",

docs/src/credits.md

@@ -1,12 +1,12 @@
-# Developer guidelines 
+# Developer guidelines
 
 This page is intended for people who wish to contribute to PlantSimEngine, and indicates the various parts to bear in mind when adding in new code.
 
 ## Working on PlantSimEngine
 
 Instructions are no different than for any other package. Use git to clone the repository [https://github.com/VirtualPlantLab/PlantSimEngine.jl](https://github.com/VirtualPlantLab/PlantSimEngine.jl).
 
-When testing your changes, your environement will need to use a command such as `Pkg.develop("PlantSimEngine")` to make use of your code. 
+When testing your changes, your environement will need to use a command such as `Pkg.develop("PlantSimEngine")` to make use of your code.
 
 We work with VSCode and are most comfortable with that IDE for Julia development. We mostly follow the manual's [Julia style guide](https://docs.julialang.org/en/v1/manual/style-guide/)
 
@@ -21,6 +21,7 @@ Other details and questions can be posted on our issues page, or as part of your
 ### Testing environments
 
 PlantSimEngine has several developer environements:
+
 - `/PlantSimEngine/test`, to check for non-regressions
 - `/PlantSimEngine/test/downstream`, whose folder contains a few benchmarks on PlantSimEngine, PlantBioPhysics and XPalm, run as a Github Action, to ensure changes don't cause performance regressions in packages depending on PlantSimEngine. You’ll need to have a version of those packages accessible if you wish to test them locally. Those are distinct from the Github Action that does some integration checks to ensure no unexpected breaking changes occurs.
  `/PlantSimEngine/docs`, to build the documentation. The documentation runs code, and some of the functions' documentation for the API are also tested as `jldoctest` instances
@@ -41,35 +42,34 @@ In the `/PlantSimEngine/docs` environment, run `/PlantSimEngine/docs/make.jl`. I
 
 ### Editing benchmarks
 
-⁃	If you wish for a branch to be benchmarked after every commit, then you need to declare it in the Github Action for benchmarks's yml file : [https://github.com/VirtualPlantLab/PlantSimEngine.jl/blob/main/.github/workflows/benchmarks_and_downstream.yml](https://github.com/VirtualPlantLab/PlantSimEngine.jl/blob/main/.github/workflows/benchmarks_and_downstream.yml) and add your branch to the `on: push:` section.
-⁃	You can view benchmarks here: https://virtualplantlab.github.io/PlantSimEngine.jl/dev/bench/index.html. They are still somewhat WIP and not yet battle-tested.
-⁃	You may occasionally need to update or delete a benchmark, in which case you will need to manually delete it in the **gh-pages** branch, in `dev/bench/index.html`
-⁃	The actual benchmark list is located in the `test/downstream` folder.
+⁃ If you wish for a branch to be benchmarked after every commit, then you need to declare it in the Github Action for benchmarks's yml file : [https://github.com/VirtualPlantLab/PlantSimEngine.jl/blob/main/.github/workflows/benchmarks_and_downstream.yml](https://github.com/VirtualPlantLab/PlantSimEngine.jl/blob/main/.github/workflows/benchmarks_and_downstream.yml) and add your branch to the `on: push:` section.
+⁃ You can view benchmarks here: <https://virtualplantlab.github.io/PlantSimEngine.jl/dev/bench/index.html>. They are still somewhat WIP and not yet battle-tested.
+⁃ You may occasionally need to update or delete a benchmark, in which case you will need to manually delete it in the **gh-pages** branch, in `dev/bench/index.html`
+⁃ The actual benchmark list is located in the `test/downstream` folder.
 
 ## Things to keep an eye out for
 
-### Downstream tests
+### Check downstream tests
 
-⁃	If your changes affect the API, then they might affect a package depending on PlantSimEngine. Benchmarks can be a way to check, as some benchmarks run other packages. Otherwise, a specific GitHub action, [https://github.com/VirtualPlantLab/PlantSimEngine.jl/blob/main/.github/workflows/Integration.yml] runs other packages’ test suites. If this action fails, then it is likely some breaking change was introduced that hasn’t been accounted for in the downstream package. If you expected a breaking change and labelled your release as such, there will be no action failure
-⁃	Note that those tests don’t build the doc (iirc), so they don’t cover that.
-⁃	API changes can also affect downstream packages’ documentation and tests...
+⁃ If your changes affect the API, then they might affect a package depending on PlantSimEngine. Benchmarks can be a way to check, as some benchmarks run other packages. Otherwise, a specific GitHub action, [https://github.com/VirtualPlantLab/PlantSimEngine.jl/blob/main/.github/workflows/Integration.yml] runs other packages’ test suites. If this action fails, then it is likely some breaking change was introduced that hasn’t been accounted for in the downstream package. If you expected a breaking change and labelled your release as such, there will be no action failure
+⁃ Note that those tests don’t build the doc (iirc), so they don’t cover that.
+⁃ API changes can also affect downstream packages’ documentation and tests...
 
 ### Which documentation pages may be affected by changes
 
 You may impact several specific documentation pages depending on what you changed. Features and API changes affect whatever they might affect, but there are some less obvious ramifications:
 
-⁃	Improving user errors may impact the **Troubleshooting** page.
-⁃	Extra features might also expand the **Tips and workarounds** page, as well as the ‘implicit contracts’ page.
-⁃	Some experimental features might be worth documenting in the dedicated **API** page, once it's added
-⁃	The roadmap "**Planned features**" page needs updating
-⁃	Potentially, other pages such as the **Credits** page, **Key Concepts**, etc. If the API makes use of new Julia features or syntax, the **Julia basics** page is probably also worth updating.
-⁃	New examples are worth making doctests of.
+⁃ Improving user errors may impact the **Troubleshooting** page.
+⁃ Extra features might also expand the **Tips and workarounds** page, as well as the ‘implicit contracts’ page.
+⁃ Some experimental features might be worth documenting in the dedicated **API** page, once it's added
+⁃ The roadmap "**Planned features**" page needs updating
+⁃ Potentially, other pages such as the **Credits** page, **Key Concepts**, etc. If the API makes use of new Julia features or syntax, the **Julia basics** page is probably also worth updating.
+⁃ New examples are worth making doctests of.
 
 ### Previewing documentation
 
 You can preview generated documentation (assuming it was able to build) relating to your PR (example given with #128) by checking the related link: [https://virtualplantlab.github.io/PlantSimEngine.jl/previews/PR128/](https://virtualplantlab.github.io/PlantSimEngine.jl/previews/PR128/)
 
-
 ## Checklist before submitting PRs
 
 ⁃ Ensure your code, uh, works
@@ -80,12 +80,12 @@ You can preview generated documentation (assuming it was able to build) relating
 ⁃ Build the PSE doc and update whatever doc tests were broken
 ⁃ Push your commit, and let the Github Actions run their course
 ⁃ Check the 'CI' GitHub action and fix if necessary
-⁃ Check downstream and benchmark GitHub actions: 
-    	If benchmarks tanked, then fix your code. If you need to add/update/delete benchmarks, do so.
-    	If you broke an integration/downstream test, you’ll need to investigate it
-    	If API changes were made, also check downstream packages’ documentation
+⁃ Check downstream and benchmark GitHub actions:
+    - If benchmarks tanked, then fix your code. If you need to add/update/delete benchmarks, do so.
+    - If you broke an integration/downstream test, you’ll need to investigate it
+    - If API changes were made, also check downstream packages’ documentation
 
-It’s probably now safe to request a merge. 
+It’s probably now safe to request a merge.
 
 ### A few extra things worth doing
 
@@ -102,11 +102,11 @@ It’s probably now safe to request a merge.
 
 ### Automatic model generation
 
-A specific feature requires generating models on the fly, to enable passing vectors to `Status` objects in multi-scale simulations. There may be more features that wish to generate models. 
+A specific feature requires generating models on the fly, to enable passing vectors to `Status` objects in multi-scale simulations. There may be more features that wish to generate models.
 
 The solution makes use of a somewhat brittle feature, `eval()`, with some subtleties. You can read more about the related world age problem [here](https://arxiv.org/abs/2010.07516), or [here](https://discourse.julialang.org/t/world-age-problem-explanation/9714/15).
 
-The related file is `model_generation_from_status_vectors.jl`, which has some additional comments. 
+The related file is `model_generation_from_status_vectors.jl`, which has some additional comments.
 
 What is important to bear in mind, is that if you call functions which generate models via `eval()`, you will need to return to top-level scope for those changes to become visible. You can see an example in `tests/helper_functions.jl` with the functions `test_filtered_output_begin` and `test_filtered_output`. The first function calls `modellist_to_mapping`, which creates some models on the fly to convert status vectors between a ModelList and its equivalent pseudo-multiscale mapping. The function is split in two so that it is possible to return to global scope and make the `eval()` changes publicly available. The second function then is able to run the simulations on the mapping with its generated models, and complete the test successfully.
 
@@ -122,4 +122,4 @@ Not all combinations of weather data structure/weather dataset size/status sizes
 
 They were briefly mentioned earlier in the page, but the test banks to increase the number of combinations tested for in terms of weather data, modellists/mappings and tracked outputs, could definitely be improved upon.
 
-TODO extra section on memory allocations, type stability etc.
+Some additional work and tests regarding tracking memory allocations, type stability etc. would also be worth implementing/documenting.

docs/src/developers.md

@@ -1,10 +1,10 @@
-# Model execution 
+# Model execution
 
-## Simulation order 
+## Simulation order
 
 `PlantSimEngine.jl` uses the [`ModelList`](@ref) to automatically compute a dependency graph between the models and run the simulation in the correct order. When running a simulation with [`run!`](@ref), the models are then executed following this simple set of rules:
 
-1. Independent models are run first. A model is independent if it can be run independently from other models, only using initializations (or nothing). 
+1. Independent models are run first. A model is independent if it can be run independently from other models, only using initializations (or nothing).
 2. Then, models that have a dependency on other models are run. The first ones are the ones that depend on an independent model. Then the ones that are children of the second ones, and then their children ... until no children are found anymore. There are two types of children models (*i.e.* dependencies): hard and soft dependencies:
-   1. Hard dependencies are always run before soft dependencies. A hard dependency is a model that list dependencies in their own method for `dep`. See [this example](https://github.com/VirtualPlantLab/PlantSimEngine.jl/blob/3d91bb053ddbd087d38dcffcedd33a9db35a0fcc/examples/dummy.jl#L39) that shows `Process2Model` defining a hard dependency on any model that simulate `process1`. Inner hard dependency graphs (*i.e.* consecutive hard-dependency children) are considered as a single soft dependency.
-   2. Soft dependencies are then run sequentially. A model has a soft dependency on another model if one or more of its inputs is computed by another model. If a soft dependency has several parent nodes (*e.g.* two different models compute two inputs of the model), it is run only if all its parent nodes have been run already. In practice, when we visit a node that has one of its parent that did not run already, we stop the visit of this branch. The node will eventually be visited from the branch of the last parent that was run.
+   1. Hard dependencies are always run before soft dependencies. A hard dependency is a model that is directly called by another model. It is declared as such by its parent that lists its hard-dependencies as `dep`. See [this example](https://github.com/VirtualPlantLab/PlantSimEngine.jl/blob/3d91bb053ddbd087d38dcffcedd33a9db35a0fcc/examples/dummy.jl#L39) that shows `Process2Model` defining a hard dependency on any model that simulates `process1`.
+   2. Soft dependencies are then run sequentially. A model has a soft dependency on another model if one or more of its inputs is computed by another model. If a soft dependency has several parent nodes (*e.g.* two different models compute two inputs of the model), it is run only if all its parent nodes have been run already. In practice, when we visit a node that has one of its parent that did not run already, we stop the visit of this branch. The node will eventually be visited from the branch of the last parent that was run.

docs/src/model_execution.md

@@ -4,13 +4,13 @@
 
 ### Varying timesteps
 
-Currently, all models are required to make use of the same timestep. Some physiological phenomenae within a plant tend to run on an hourly basis, others are slower. Weather data is often provided daily. Enabling different timesteps depending on the model is on the roadmap.
+Currently, all models are required to make use of the same timestep. Some physiological phenomenae within a plant tend to run on an hourly basis, others are slower. Weather data is often provided daily. Enabling different timesteps depending on the model is on the roadmap, and is planned as the next milestone.
 
 ### Multi-plant/Multi-species simulations
 
 A goal for PlantSimEngine down the line is to be able to simulate complex scenes with data comprising several plants, possibly of different species, for agroforestry purposes.
 
-Its current state doesn't enable practical declaration of several plant species, or multiple plants relying on similar subsets of models with partially different models or parameters. 
+Its current state doesn't enable practical declaration of several plant species, or multiple plants relying on similar subsets of models with partially different models or parameters.
 
 ## Minor features
 
@@ -34,23 +34,22 @@ Its current state doesn't enable practical declaration of several plant species,
 
 ## Possible features (likely not a priority)
 
-- API enabling iterative builds and validation of mappings and modellists
-- Improved parallelisation 
+- API enabling iterative builds and validation of mappings and ModelLists
+- Build step for the models, *i.e.* a function that would write a mapping or ModelList into a Julia script for validation, improved readability and (maybe) performance (no need to traverse the dependency graph anymore).
+- Improved parallelisation
 - Reintroduce multi-object parallelisation in single-scale
 
 ## Other minor points
 
 - Examples/solutions for floating-point accumulation errors
 - More examples for fitting/type conversion/error propagation
-- MTG couple of new features #106 
+- MTG couple of new features #106
 - Other minor bugs
 - Unrolling the run! function
 
 ## Other
 
-- Reproducing another FSPM ?
-- Diffusion model example ?
+- Reproducing another FSPM?
+- Diffusion model example?
 
-The full list of issues can be found [here](https://github.com/VirtualPlantLab/PlantSimEngine.jl/issues)
-
-TODO Detail other related packages' roadmaps (mostly stuff like PlantGeom's mtg reference mesh linking) ? PBP updates ?
+The full list of issues can be found [here](https://github.com/VirtualPlantLab/PlantSimEngine.jl/issues)

docs/src/planned_features.md

@@ -52,8 +52,4 @@ Also of importance:
 
 Many of these are also briefly presented in [this Julia Data Science](https://juliadatascience.io/julia_basics) guide, which also happens to focus on the DataFrames.jl package.
 
-Understanding more about methods, parametric types and the typing system is usually worthwhile, when working with Julia packages.
-
-TODO point to Rémi's videos ? Other videos ?
-TODO extra concepts useful for developers ?
-
+Understanding more about methods, parametric types and the typing system is usually worthwhile, when working with Julia packages.