diff --git a/docs/toc.yml b/docs/toc.yml index 21e973c..80d5142 100644 --- a/docs/toc.yml +++ b/docs/toc.yml @@ -2,8 +2,6 @@ href: ../README.md - name: Linear Dynamical Systems -- name: Getting Started - href: ../examples/LinearDynamicalSystems/README.md - name: Kinematics - name: Simulated Data href: ../examples/LinearDynamicalSystems/Kinematics/SimulatedData/README.md @@ -11,8 +9,8 @@ href: ../examples/LinearDynamicalSystems/Kinematics/ZebrafishCentroidTracking/README.md - name: Foraging Mouse href: ../examples/LinearDynamicalSystems/Kinematics/ForagingMouse/README.md -- name: Optimizing Model Parameters for Foraging Mouse - href: ../examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/README.md +- name: Online Learning and Kinematics Inference of a Foraging Mouse + href: ../examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/README.md - name: Forecasting Foraging Mouse Kinematics href: ../examples/LinearDynamicalSystems/Kinematics/ForecastingForagingMouse/README.md - name: Linear Regression @@ -20,10 +18,8 @@ href: ../examples/LinearDynamicalSystems/LinearRegression/SimulatedData/README.md - name: Receptive Field Simple Cell href: ../examples/LinearDynamicalSystems/LinearRegression/ReceptiveFieldSimpleCell/README.md - + - name: Hidden Markov Models -- name: Getting Started - href: ../examples/HiddenMarkovModels/README.md - name: Simulated Data href: ../examples/HiddenMarkovModels/SimulatedData/README.md - name: Inferring Behavioral State from Kinematics @@ -52,5 +48,5 @@ href: ../examples/PointProcessDecoder/DecodePositionFromHippocampusClusterless/README.md - name: Decode Position from Striatum using Sorted Spikes href: ../examples/PointProcessDecoder/DecodePositionFromStriatumSortedUnits/README.md -- name: Classify Dynamics and Decode Position from Hippocampal Sorted Units - href: ../examples/PointProcessDecoder/ClassifyPositionFromHippocampusSortedUnits/README.md \ No newline at end of file +- name: Classify Dynamics and Decode Position from Hippocampal Sorted Units + href: ../examples/PointProcessDecoder/ClassifyPositionFromHippocampusSortedUnits/README.md diff --git a/examples/HiddenMarkovModels/ExtendedModelConfiguration/README.md b/examples/HiddenMarkovModels/ExtendedModelConfiguration/README.md index bdfcc22..62bfb2e 100644 --- a/examples/HiddenMarkovModels/ExtendedModelConfiguration/README.md +++ b/examples/HiddenMarkovModels/ExtendedModelConfiguration/README.md @@ -9,7 +9,7 @@ In the following example, you can see how to construct a Hidden Markov Model (HM Ensure you have Bonsai 2.9 installed on your system. You will also need to have the [uv Python environment manager](https://docs.astral.sh/uv/) installed. > [!NOTE] -> The `ssm` package uses an old version of `setuptools` that is incompatible with modern package management tools. Because of this, you first need to create a new Python virtual environment using `uv venv`, install the required packages using `uv pip install -r requirements.txt`, and then synchronize the Python environment using `uv sync --no-build-isolation`. +> The `ssm` package uses an old version of `setuptools` that is incompatible with modern package management tools. Because of this, you first need to create a new Python virtual environment using `uv venv`, install the required packages using `uv pip install -r requirements.txt`, and then synchronize the Python environment using `uv sync --no-build-isolation`. ### Save a Custom HMM Configuration @@ -27,10 +27,10 @@ When the workflow is run, the output of the `ConfigureHMM` node is a custom HMM ### Load a Custom HMM Configuration -This example workflow demonstrates how to load the custom HMM model saved in the previous workflow. +This example workflow demonstrates how to load the custom HMM model saved in the previous workflow. :::workflow ![Hidden Markov Models - Loading Custom Model From Disk](LoadModelConfig.bonsai) ::: -The `LoadStateParameters` is a `SelectMany` operator which will load the JSON file specified in the `Path` property, and outputs a `ModelParameters` object. The `Dimensions`, `NumStates`, and `StateParameters` properties of the `ModelParameters` object are then used as input parameters to the `CreateHMM` operator, which creates an HMM model with the parameters loaded from the JSON file. \ No newline at end of file +The `LoadStateParameters` is a `SelectMany` operator which will load the JSON file specified in the `Path` property, and outputs a `ModelParameters` object. The `Dimensions`, `NumStates`, and `StateParameters` properties of the `ModelParameters` object are then used as input parameters to the `CreateHMM` operator, which creates an HMM model with the parameters loaded from the JSON file. diff --git a/examples/HiddenMarkovModels/InferringBehavioralStateFromKinematics/README.md b/examples/HiddenMarkovModels/InferringBehavioralStateFromKinematics/README.md index c843c1c..775a0b7 100644 --- a/examples/HiddenMarkovModels/InferringBehavioralStateFromKinematics/README.md +++ b/examples/HiddenMarkovModels/InferringBehavioralStateFromKinematics/README.md @@ -9,25 +9,10 @@ In the following example, you can see how the Hidden Markov Model (HMM) can be u Ensure you have Bonsai 2.9 installed on your system. You will also need to have the [uv Python environment manager](https://docs.astral.sh/uv/) installed. > [!NOTE] -> The `ssm` package uses an old version of `setuptools` that is incompatible with modern package management tools. Because of this, you first need to create a new Python virtual environment using `uv venv`, install the required packages using `uv pip install -r requirements.txt`, and then synchronize the Python environment using `uv sync --no-build-isolation`. - -### Dependencies - -If you used the bootstrapping method, you don't have to worry about the package dependencies, as these should be already installed. However, if creating a new environment or integrating into an existing one, you will need to install the following packages: - -* Bonsai - Core v2.8.1 -* Bonsai - Design v2.8.0 -* Bonsai - Editor v2.8.0 -* Bonsai - Expression Scripting v2.8.0 -* Bonsai - Gui v0.1.0 -* Bonsai - ML Hidden Markov Models v0.2.0 -* Bonsai - ML Linear Dynamical Systems v0.2.0 -* Bonsai - ML Visualizers Hidden Markov Models v0.2.0 -* Bonsai - ML Visualizers Linear Dynamical Systems v0.2.0 -* Bonsai - Numerics v0.9.0 +> The `ssm` package uses an old version of `setuptools` that is incompatible with modern package management tools. Because of this, you first need to create a new Python virtual environment using `uv venv`, install the required packages using `uv pip install -r requirements.txt`, and then synchronize the Python environment using `uv sync --no-build-isolation`. > [!WARNING] -> This example builds on the [LDS Kinematics Foraging Mouse example](../../LinearDynamicalSystems/Kinematics/ForagingMouse/README.md) and requires both the *Bonsai.ML.LinearDynamicalSystems* package and the *Bonsai.ML.HiddenMarkovModels* package. See the installation guide to ensure both packages are installed and working correctly. +> This example builds on the [LDS Kinematics Foraging Mouse example](../../LinearDynamicalSystems/Kinematics/ForagingMouse/README.md) and requires both the _Bonsai.ML.LinearDynamicalSystems_ package and the _Bonsai.ML.HiddenMarkovModels_ package. See the installation guide to ensure both packages are installed and working correctly. ### Workflow @@ -37,7 +22,7 @@ Below is the workflow. ![Hidden Markov Models - Inferring Behavioral State](InferringBehavioralState.bonsai) ::: -In this example, a Hidden Markov Model (HMM) is used to infer the hidden behavioral state of a foraging mouse. The workflow creates a python runtime (`CreateRuntime`), and then loads both the HMM module (`LoadHMMModule`) and the LDS module (`LoadLDSModule`). The `MouseTracking` group workflow performs image processing to calculate the mouse's `Centroid` position. The `Centroid` is used with `InputMapping` to initialize a Kalman Filter (KF) Kinematics model (`CreateKFModel`) with the initial parameters `Position_x0` and `Position_y0` set to the X and Y values of the `Centroid`. We set the other parameters of the KF model to be the parameters that we think will best track the animal, though these parameters can also be learned online [see the ModelOptimizationForagingMouse example](../../LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/README.md). +In this example, a Hidden Markov Model (HMM) is used to infer the hidden behavioral state of a foraging mouse. The workflow creates a python runtime (`CreateRuntime`), and then loads both the HMM module (`LoadHMMModule`) and the LDS module (`LoadLDSModule`). The `MouseTracking` group workflow performs image processing to calculate the mouse's `Centroid` position. The `Centroid` is used with `InputMapping` to initialize a Kalman Filter (KF) Kinematics model (`CreateKFModel`) with the initial parameters `Position_x0` and `Position_y0` set to the X and Y values of the `Centroid`. We set the other parameters of the KF model to be the parameters that we think will best track the animal, though these parameters can also be learned online [see the OnlineLearningAndKinematicsInference example](../../LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/README.md). We are going to use the absolute velocity and acceleration of the animal as observations to the HMM model. We are going to use 6 hidden states for our model. When we initialize our HMM using the `CreateHMM` node, we set the `Dimensions` to 2, for the number of features of our observations, and the `NumStates` to 6, for the number of hidden behavioral states. The `Observations` and `Transitions` parameters will be left to their default values, `Gaussian` and `Stationary`, respectively. diff --git a/examples/HiddenMarkovModels/README.md b/examples/HiddenMarkovModels/README.md deleted file mode 100644 index 1b72f23..0000000 --- a/examples/HiddenMarkovModels/README.md +++ /dev/null @@ -1,129 +0,0 @@ -# Getting Started - -In general, when using the HiddenMarkovModels package, you need to configure your Bonsai workflow to run the correct python environment which has the [ssm](https://github.com/lindermanlab/ssm) package installed. - -If you followed the installation guide, you will have a basic Bonsai-python environment for using the LinearDynamicalSystems package. You will also have installed the basic dependencies required to build and run the Bonsai workflows. However, if you wish to run the examples, we highly recommend following this guide to create and configure the environments needed for each example. This guide has been made for users starting from scratch to get started. Alternatively, you can use the environment you created in the previous guide and install the necessary package dependencies required to run each workflow. - -All of the examples are self-contained and can be found by going to the [Bonsai.ML - Examples](https://github.com/bonsai-rx/machinelearning-examples) repo. For each example, you can bootstrap the python and Bonsai environments from within each example directory. Below, we demonstrate how to get started using the examples by bootstrapping the example environments that are needed to run the example workflow. - -### Windows - -#### Dependencies - -You must have the following installed on your PC in order to run the example workflows: - -- [Python (v3.10)](https://www.python.org/downloads/) *Note: python must be installed using the windows installer and must be added to the system PATH* -- [Git](https://git-scm.com/downloads) - -#### Instructions - -Open the terminal or powershell. Start by cloning the [Bonsai.ML - Examples](https://github.com/bonsai-rx/machinelearning-examples) repo with: - -```cmd -git clone https://github.com/bonsai-rx/machinelearning-examples.git -``` - -Change directory to one of the examples: - -```cmd -cd .\machinelearning-examples\examples\HiddenMarkovModels\ForagingMouse -``` - -To create the python virtual environment and install the package, run the following: - -```cmd -python -m venv .venv -.\.venv\Scripts\activate -pip install numpy cython -pip install ssm@git+https://github.com/lindermanlab/ssm@6c856ad3967941d176eb348bcd490cfaaa08ba60 -``` - -> [!WARNING] -> If you have compatibility issues with the latest version of numpy (v2.0), try the following: -> ```cmd -> pip install numpy==1.26.4 -> ``` - -To create the bonsai environment and install the packages, run the powershell script: - -```cmd -cd .bonsai -powershell .\Setup.ps1 -``` - -Once installed, run the bonsai executable with: - -```cmd -.\Bonsai.exe -``` - -Open the workflow example and start the bonsai workflow. - -### Linux - -#### Notes on Running Bonsai in Linux - -Currently, the examples have only been tested on Ubuntu 22.04. Running Bonsai on Linux is still being tested and should be used with caution. We cannot guarantee that these instructions will work for all Linux distributions or versions. It is important that you consult the general [Bonsai installation guide on Linux](https://github.com/orgs/bonsai-rx/discussions/1101) to ensure that the underlying Bonsai package dependencies are met and installed properly. - -#### Dependencies - -You must have the following installed on your PC in order to run the example workflows: - -- [Python (v3.10)](https://www.python.org/downloads/) *comes installed with the latest version of Ubuntu 22.04* -- [Git](https://git-scm.com/downloads) -- [Mono](https://www.mono-project.com/download/stable/#download-lin) -- [OpenCV and OpenGL binaries](https://github.com/orgs/bonsai-rx/discussions/1101) - -#### Instructions - -Open up a terminal and start by cloning the [Bonsai.ML - Examples](https://github.com/bonsai-rx/machinelearning-examples) repo: - -```cmd -git clone https://github.com/bonsai-rx/machinelearning-examples.git -``` - -Change directory to one of the examples. For instance: - -```cmd -cd machinelearning-examples/examples/LinearDynamicalSystems/Kinematics/SimulatedData -``` - -Create the python virtual environment and install the lds python package with the following: - -```cmd -python3 -m venv .venv -source .venv/bin/activate -pip install numpy cython -pip install ssm@git+https://github.com/lindermanlab/ssm@6c856ad3967941d176eb348bcd490cfaaa08ba60 -``` - -> [!WARNING] -> The latest version of numpy (v2.0) is incompatible with the ssm package. To install the latest compatible version of numpy, use: -> ```cmd -> pip install numpy==1.26.4 -> ``` - -Run the `Setup.sh` script using the following: - -```cmd -cd .bonsai -./Setup.sh -``` - -> [!TIP] -> The `Setup.sh` script uses the `xmllint` and `xmlstarlet` packages to convert the assembly location paths in the bonsai config file from Windows to Linux paths. If you do not have these packages already installed on your computer, you can install the `xmllint` and the `xmlstarlet` packages using: -> ```cmd -> sudo apt install -y libxml2-utils xmlstarlet -> ``` - -Once the bonsai environment has been created, activate the bonsai environment and run the Bonsai executable: - -```cmd -source ./activate -bonsai -``` - -> [!TIP] -> If your desktop theme is set to dark mode, the Bonsai GUI can display in weird ways due to the way that mono tries to use the desktop theme in applications. If you use the `bonsai-clean` command instead of the `bonsai` command, this will reset the theme that mono uses to the default theme, which can imporove the appearance of the Bonsai GUI. - -Open the workflow example and start the bonsai workflow. diff --git a/examples/HiddenMarkovModels/SimulatedData/README.md b/examples/HiddenMarkovModels/SimulatedData/README.md index fb13ec0..354d23c 100644 --- a/examples/HiddenMarkovModels/SimulatedData/README.md +++ b/examples/HiddenMarkovModels/SimulatedData/README.md @@ -9,21 +9,7 @@ In the following example, you can see how the Hidden Markov Model (HMM) can be u Ensure you have Bonsai 2.9 installed on your system. You will also need to have the [uv Python environment manager](https://docs.astral.sh/uv/) installed. > [!NOTE] -> The `ssm` package uses an old version of `setuptools` that is incompatible with modern package management tools. Because of this, you first need to create a new Python virtual environment using `uv venv`, install the required packages using `uv pip install -r requirements.txt`, and then synchronize the Python environment using `uv sync --no-build-isolation`. - -### Dependencies - -If you used the bootstrapping method, you don't have to worry about the package dependencies, as these should be already installed. However, if creating a new environment or integrating into an existing one, you will need to install the following packages: - -* Bonsai - Core v2.8.1 -* Bonsai - Design v2.8.0 -* Bonsai - Editor v2.8.0 -* Bonsai - Gui Zedgraph v0.1.1 -* Bonsai - ML v0.2.0 -* Bonsai - ML Hidden Markov Models v0.2.0 -* Bonsai - ML Visualizers Hidden Markov Models v0.2.0 -* Bonsai - Numerics v0.9.0 -* Bonsai - Scripting Python v0.2.0 +> The `ssm` package uses an old version of `setuptools` that is incompatible with modern package management tools. Because of this, you first need to create a new Python virtual environment using `uv venv`, install the required packages using `uv pip install -r requirements.txt`, and then synchronize the Python environment using `uv sync --no-build-isolation`. ### Workflow @@ -35,7 +21,7 @@ Below is the workflow. In this example, a Hidden Markov Model (HMM) is used to infer the hidden state of a simulated system. The workflow creates a python runtime (`CreateRuntime`), loads the HMM module (`LoadHMMModule`), and then initializes an HMM model with two states (`CreateHMM`). The simulation, encapsulated by a group workflow (`SimulatedDataWith2States`), generates data from two distinct multivariate normal distributions. The distribution that the simulation draws samples from switches at a specified rate. These `Data` are then formatted into a string representation of a python list, and subsequently fed as `Observation`s to the HMM model. The HMM model performs inference of the hidden state (`InferState`) given the observation. -When the HMM model is created, the parameters are not yet fit to the data. After 80 data points, the mini batch of data is sent to the model to be fit to the data (`RunFitAsync`). Once the fitting procedure completes asynchronously, the model's parameters will be better tuned to the system, and the model begins performing inferrence of the state of the simulated system. +When the HMM model is created, the parameters are not yet fit to the data. After 80 data points, the mini batch of data is sent to the model to be fit to the data (`RunFitAsync`). Once the fitting procedure completes asynchronously, the model's parameters will be better tuned to the system, and the model begins performing inferrence of the state of the simulated system. To visualize this, the `SimulationVisualizer` group node displays two graphs. On the left, the X and Y data samples are plotted as individual points in the scatter plot. After a few samples, you should start to see two distinct clusters emerge. On the right, you will see the inferred state of the model, represented as the probability of being in a particular state. After the model has been fit, you should see that the inferred state aligns with the cluster that data samples are being drawn from. diff --git a/examples/LinearDynamicalSystems/Kinematics/ForagingMouse/README.md b/examples/LinearDynamicalSystems/Kinematics/ForagingMouse/README.md index 9cb82e8..cca75f1 100644 --- a/examples/LinearDynamicalSystems/Kinematics/ForagingMouse/README.md +++ b/examples/LinearDynamicalSystems/Kinematics/ForagingMouse/README.md @@ -8,21 +8,6 @@ In the following example, you can find how the Kalman Filter can be used to infe Ensure you have Bonsai 2.9 installed on your system. You will also need to have the [uv Python environment manager](https://docs.astral.sh/uv/) installed. You can bootstrap the Python environment by running `uv sync` in the example directory. -### Dependencies - -If you used the bootstrapping method, you dont have to worry about the package dependencies, as these should be already installed. However, if creating a new environment or integrating into an existing one, you will need to install the following packages: - -* Bonsai - Core v2.8.1 -* Bonsai - Design v2.8.0 -* Bonsai - Editor v2.8.0 -* Bonsai - ML v0.1.0 -* Bonsai - ML LinearDynamicalSystems v0.1.0 -* Bonsai - ML Visualizers v0.1.0 -* Bonsai - Scripting v2.8.0 -* Bonsai - Scripting Python v0.2.0 -* Bonsai - Vision v2.8.1 -* Bonsai - Vision Design v2.8.1 - ### Dataset The video for this example was generously provided by the Sainsbury Wellcome Centre Foraging Behaviour Working Group. (2023). Aeon: An open-source platform to study the neural basis of ethological behaviours over naturalistic timescales, [https://doi.org/10.5281/zenodo.8413142](https://doi.org/10.5281/zenodo.8413142) @@ -37,7 +22,7 @@ Below is the workflow for inferring kinematics of a mouse during a foraging task ![Kinematics - Mouse Foraging Behaviour](ForagingMouse.bonsai) ::: -In this example, a Kalman Filter is used to infer the position, velocity, and acceleration of the freely moving mouse. The frames per second of the video is 50 Hz, so we set the `Fps` property of the `CreateKFModel` node to 50. The workflow performs centroid tracking inside of the `MouseTracking` group node, which uses a standard image analysis pipeline to extract the `Centroid` of the animal. The `Centroid` data is then converted into a type of `Observation2D` that the model then uses to perform inference using the `PerformInference` node. +In this example, a Kalman Filter is used to infer the position, velocity, and acceleration of the freely moving mouse. The frames per second of the video is 50 Hz, so we set the `Fps` property of the `CreateKFModel` node to 50. The workflow performs centroid tracking inside of the `MouseTracking` group node, which uses a standard image analysis pipeline to extract the `Centroid` of the animal. The `Centroid` data is then converted into a type of `Observation2D` that the model then uses to perform inference using the `PerformInference` node. Since the tracking pipeline takes some time to initialize, we wait to create the kalman filter using the `SubscribeWhen` node only once `TrackingStarted` has produced a value, which is conditioned on centroid != NaN. diff --git a/examples/LinearDynamicalSystems/Kinematics/ForecastingForagingMouse/README.md b/examples/LinearDynamicalSystems/Kinematics/ForecastingForagingMouse/README.md index 22a5a66..966acb0 100644 --- a/examples/LinearDynamicalSystems/Kinematics/ForecastingForagingMouse/README.md +++ b/examples/LinearDynamicalSystems/Kinematics/ForecastingForagingMouse/README.md @@ -8,21 +8,6 @@ In the following example, you can see how the Kalman Filter can be use to foreca Ensure you have Bonsai 2.9 installed on your system. You will also need to have the [uv Python environment manager](https://docs.astral.sh/uv/) installed. You can bootstrap the Python environment by running `uv sync` in the example directory. -### Dependencies - -If you used the environment bootstrapping method, you dont have to worry about the package dependencies, as these should be already installed. However, if creating a new environment or integrating into an existing one, you will need to install the following packages: - -* Bonsai - Core v2.8.1 -* Bonsai - Design v2.8.0 -* Bonsai - Editor v2.8.0 -* Bonsai - ML v0.2.0 -* Bonsai - ML LinearDynamicalSystems v0.2.0 -* Bonsai - ML Visualizers v0.2.0 -* Bonsai - Scripting v2.8.0 -* Bonsai - Scripting Python v0.2.0 -* Bonsai - Vision v2.8.1 -* Bonsai - Vision Design v2.8.1 - ### Dataset The video for this example was generously provided by the Sainsbury Wellcome Centre Foraging Behaviour Working Group. (2023). Aeon: An open-source platform to study the neural basis of ethological behaviours over naturalistic timescales, [https://doi.org/10.5281/zenodo.8413142](https://doi.org/10.5281/zenodo.8413142) diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/Bonsai.config b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/Bonsai.config similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/Bonsai.config rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/Bonsai.config diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/NuGet.config b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/NuGet.config similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/NuGet.config rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/NuGet.config diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/Setup.ps1 b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/Setup.ps1 similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/Setup.ps1 rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/Setup.ps1 diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/Setup.sh b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/Setup.sh similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/Setup.sh rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/Setup.sh diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/activate b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/activate similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/activate rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/activate diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/deactivate b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/deactivate similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/deactivate rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/deactivate diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/run b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/run similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/.bonsai/run rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/.bonsai/run diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/ComparingModelOutputs.gif b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/ComparingModelOutputs.gif similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/ComparingModelOutputs.gif rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/ComparingModelOutputs.gif diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/ModelOptimizationForagingMouse.bonsai b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/OnlineLearningAndKinematicsInference.bonsai similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/ModelOptimizationForagingMouse.bonsai rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/OnlineLearningAndKinematicsInference.bonsai diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/README.md b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/README.md similarity index 81% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/README.md rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/README.md index b3bf97b..0ea7d3f 100644 --- a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/README.md +++ b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/README.md @@ -1,4 +1,4 @@ -# Optimizing Model Parameters for Foraging Mouse Example +# Online Learning and Kinematics Inference in Foraging Mice The code for this repo can be found [here](https://github.com/bonsai-rx/machinelearning-examples/tree/main/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse). @@ -8,21 +8,6 @@ In the following example, you can see how the parameters of the Kalman Filter ca Ensure you have Bonsai 2.9 installed on your system. You will also need to have the [uv Python environment manager](https://docs.astral.sh/uv/) installed. You can bootstrap the Python environment by running `uv sync` in the example directory. -### Dependencies - -If you used the environment bootstrapping method, you dont have to worry about the package dependencies, as these should be already installed. However, if creating a new environment or integrating into an existing one, you will need to install the following packages: - -* Bonsai - Core v2.8.1 -* Bonsai - Design v2.8.0 -* Bonsai - Editor v2.8.0 -* Bonsai - ML v0.2.0 -* Bonsai - ML LinearDynamicalSystems v0.2.0 -* Bonsai - ML Visualizers v0.2.0 -* Bonsai - Scripting v2.8.0 -* Bonsai - Scripting Python v0.2.0 -* Bonsai - Vision v2.8.1 -* Bonsai - Vision Design v2.8.1 - ### Dataset The video for this example was generously provided by the Sainsbury Wellcome Centre Foraging Behaviour Working Group. (2023). Aeon: An open-source platform to study the neural basis of ethological behaviours over naturalistic timescales, [https://doi.org/10.5281/zenodo.8413142](https://doi.org/10.5281/zenodo.8413142) @@ -34,10 +19,10 @@ You can download the `ForagingMouseExampleVid.avi` video file here: [https://doi Below is the workflow. It is similar to the workflow that was used to infer the kinematics of a mouse during foraging (see example here)[https://github.com/bonsai-rx/machinelearning-examples/tree/main/examples/LinearDynamicalSystems/Kinematics/ForagingMouse] with the addition of a few nodes that will run optimization of the models parameters over small batches of data that are collected continuously. :::workflow -![Model Optimization for Mouse Foraging Behaviour](ModelOptimizationForagingMouse.bonsai) +![Online Learning and Kinematics Inference of a Foraging Mouse](OnlineLearningAndKinematicsInference.bonsai) ::: -You can find specific information about the tracking workflow and model inference workflow by going to the Foraging Mouse example. In addition to the tracking workflow and model inference, only a few nodes are needed to perform optimization of the models parameters. There is a specific branch of the workflow where the `Observation` subject is subscribed to after the `ModelParams` fires, and gets passed to a `RunOptimization` node. This node has a number of properties which me be useful, such as the batch size of the training data, which parameters to optimize, etc. Once the data buffer is full, the model runs optimization asynchronously. You should see in the beginning that the model will tend to overshoot the position of the mouse, since the parameters are not well adjusted. Over time, the output of the model will improve and the inference of the model will more closely track the position of the mouse with multiple learning iterations. +You can find specific information about the tracking workflow and model inference workflow by going to the Foraging Mouse example. In addition to the tracking workflow and model inference, only a few nodes are needed to perform optimization of the models parameters. There is a specific branch of the workflow where the `Observation` subject is subscribed to after the `ModelParams` fires, and gets passed to a `RunOptimization` node. This node has a number of properties which me be useful, such as the batch size of the training data, which parameters to optimize, etc. Once the data buffer is full, the model runs optimization asynchronously. You should see in the beginning that the model will tend to overshoot the position of the mouse, since the parameters are not well adjusted. Over time, the output of the model will improve and the inference of the model will more closely track the position of the mouse with multiple learning iterations. The easiest way to observe the effects that optimizing the models parameters has on the models output is to compare the outputs between a model that undergoes optimization and an identical model that retains its initial parameters. With each iteration, the optimized model will diverge from the unoptimized model and more closely track the position of the mouse. This is what you would see: diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/pyproject.toml b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/pyproject.toml similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/pyproject.toml rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/pyproject.toml diff --git a/examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/uv.lock b/examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/uv.lock similarity index 100% rename from examples/LinearDynamicalSystems/Kinematics/ModelOptimizationForagingMouse/uv.lock rename to examples/LinearDynamicalSystems/Kinematics/OnlineLearningAndKinematicsInference/uv.lock diff --git a/examples/LinearDynamicalSystems/Kinematics/SimulatedData/README.md b/examples/LinearDynamicalSystems/Kinematics/SimulatedData/README.md index bc762de..e8f5b3e 100644 --- a/examples/LinearDynamicalSystems/Kinematics/SimulatedData/README.md +++ b/examples/LinearDynamicalSystems/Kinematics/SimulatedData/README.md @@ -8,23 +8,6 @@ In the following example, you can find how the Kalman Filter can be used for kin Ensure you have Bonsai 2.9 installed on your system. You will also need to have the [uv Python environment manager](https://docs.astral.sh/uv/) installed. You can bootstrap the Python environment by running `uv sync` in the example directory. -### Dependencies - -If you used the bootstrapping method, you dont have to worry about the package dependencies, as these should be already installed. However, if creating a new environment or integrating into an existing one, you will need to install the following packages: - -* Bonsai - Core v2.8.1 -* Bonsai - Design v2.8.0 -* Bonsai - Editor v2.8.0 -* Bonsai - ML v0.1.0 -* Bonsai - ML LinearDynamicalSystems v0.1.0 -* Bonsai - ML Visualizers v0.1.0 -* Bonsai - Numerics v0.9.0 -* Bonsai - Scripting v2.8.0 -* Bonsai - Scripting Python v0.2.0 -* Bonsai - System v2.8.0 -* Bonsai - Vision v2.8.1 -* Bonsai - Vision Design v2.8.0 - ### Workflow Below is the workflow for running the Kalman Filter Kinematics model on simulated data. diff --git a/examples/LinearDynamicalSystems/Kinematics/ZebrafishCentroidTracking/README.md b/examples/LinearDynamicalSystems/Kinematics/ZebrafishCentroidTracking/README.md index 30f092c..7da5ff7 100644 --- a/examples/LinearDynamicalSystems/Kinematics/ZebrafishCentroidTracking/README.md +++ b/examples/LinearDynamicalSystems/Kinematics/ZebrafishCentroidTracking/README.md @@ -8,21 +8,6 @@ In the following example, you can find how the Kalman Filter can be used to infe Ensure you have Bonsai 2.9 installed on your system. You will also need to have the [uv Python environment manager](https://docs.astral.sh/uv/) installed. You can bootstrap the Python environment by running `uv sync` in the example directory. -### Dependencies - -If you used the bootstrapping method, you dont have to worry about the package dependencies, as these should be already installed. However, if creating a new environment or integrating into an existing one, you will need to install the following packages: - -* Bonsai - Core v2.8.1 -* Bonsai - Design v2.8.0 -* Bonsai - Editor v2.8.0 -* Bonsai - ML v0.1.0 -* Bonsai - ML LinearDynamicalSystems v0.1.0 -* Bonsai - ML Visualizers v0.1.0 -* Bonsai - Scripting v2.8.0 -* Bonsai - Scripting Python v0.2.0 -* Bonsai - Vision v2.8.1 -* Bonsai - Vision Design v2.8.1 - ### Dataset The video in this dataset was provided by Nicholas Guilbeault in the Thiele lab at the University of Toronto and published as part of Guilbeault, N.C., Guerguiev, J., Martin, M. et al. BonZeb: open-source, modular software tools for high-resolution zebrafish tracking and analysis. Sci Rep 11, 8148 (2021). [https://doi.org/10.1038/s41598-021-85896-x](https://doi.org/10.1038/s41598-021-85896-x). @@ -37,7 +22,7 @@ Below is the workflow for inferring kinematics of zebrafish swimming. ![Kinematics - Zebrafish Tracking](ZebrafishTracking.bonsai) ::: -In this example, a Kalman Filter is used to infer the position, velocity, and acceleration of a freely swimming zebrafish. The original video was collected at 200 Hz, so the `Fps` property of the `CreateKFModel` node is set to 200. The workflow works by performing centroid tracking inside of the `ZebrafishTracking` node, which performs image analyses to extract the `Centroid` of the zebrafish. The `Centroid` data is then converted into a type of `Observation2D` that the model then uses to perform inference using the `PerformInference` node. +In this example, a Kalman Filter is used to infer the position, velocity, and acceleration of a freely swimming zebrafish. The original video was collected at 200 Hz, so the `Fps` property of the `CreateKFModel` node is set to 200. The workflow works by performing centroid tracking inside of the `ZebrafishTracking` node, which performs image analyses to extract the `Centroid` of the zebrafish. The `Centroid` data is then converted into a type of `Observation2D` that the model then uses to perform inference using the `PerformInference` node. Since the tracking pipeline takes some time to initialize, we wait to create the kalman filter using the `SubscribeWhen` node only once `TrackingStarted` has produced a value, which is conditioned on centroid != NaN. diff --git a/examples/LinearDynamicalSystems/LinearRegression/ReceptiveFieldSimpleCell/README.md b/examples/LinearDynamicalSystems/LinearRegression/ReceptiveFieldSimpleCell/README.md index 15d1407..03ae4b8 100644 --- a/examples/LinearDynamicalSystems/LinearRegression/ReceptiveFieldSimpleCell/README.md +++ b/examples/LinearDynamicalSystems/LinearRegression/ReceptiveFieldSimpleCell/README.md @@ -8,14 +8,6 @@ In the following example, you can find how the Kalman Filter can be used to esti Ensure you have Bonsai 2.9 installed on your system. You will also need to have the [uv Python environment manager](https://docs.astral.sh/uv/) installed. You can bootstrap the Python environment by running `uv sync` in the example directory. -### Dependencies - -If you used the bootstrapping method, you dont have to worry about the package dependencies, as these should be already installed. However, if creating a new environment or integrating into an existing one, you will need to install the following packages: - -* Bonsai - System v2.8.1 -* Bonsai - ML LinearDynamicalSystems v0.2.0 -* Bonsai - ML Visualizers v0.2.0 - ### Dataset The data for this example was generously provided by the authors of "Touryan, J., Felsen, G., & Dan, Y. (2005). Spatial structure of complex cell receptive fields measured with natural images. Neuron, 45(5), 781-791." diff --git a/examples/LinearDynamicalSystems/LinearRegression/SimulatedData/README.md b/examples/LinearDynamicalSystems/LinearRegression/SimulatedData/README.md index c1846af..b71b2a5 100644 --- a/examples/LinearDynamicalSystems/LinearRegression/SimulatedData/README.md +++ b/examples/LinearDynamicalSystems/LinearRegression/SimulatedData/README.md @@ -8,14 +8,6 @@ In the following example, you will see how the Kalman Filter can be used to perf Ensure you have Bonsai 2.9 installed on your system. You will also need to have the [uv Python environment manager](https://docs.astral.sh/uv/) installed. You can bootstrap the Python environment by running `uv sync` in the example directory. -### Dependencies - -If you used the bootstrapping method, you dont have to worry about the package dependencies, as these should be already installed. However, if creating a new environment or integrating into an existing one, you will need to install the following packages: - -* Bonsai - System v2.8.1 -* Bonsai - ML LinearDynamicalSystems v0.2.0 -* Bonsai - ML Visualizers v0.2.0 - ### Workflow Below is the example workflow for how to perform bayesian linear regression with noisy data generated from a simple linear model. diff --git a/examples/LinearDynamicalSystems/README.md b/examples/LinearDynamicalSystems/README.md deleted file mode 100644 index e060f00..0000000 --- a/examples/LinearDynamicalSystems/README.md +++ /dev/null @@ -1,138 +0,0 @@ -# Getting Started - -In general, when using the LinearDynamicalSystems package, you need to configure your Bonsai workflow to run the correct python environment which has the [lds_python](https://github.com/joacorapela/lds_python) package installed. - -If you followed the installation guide, you will have a basic Bonsai-python environment for using the LinearDynamicalSystems package. You will also have installed the basic dependencies required to build and run the Bonsai workflows. However, if you wish to run the examples, we highly recommend following this guide to create and configure the environments needed for each example. This guide has been made for users starting from scratch to get started. Alternatively, you can use the environment you created in the previous guide and install the necessary package dependencies required to run each workflow. - -All of the examples are self-contained and can be found by going to the [Bonsai.ML - Examples](https://github.com/bonsai-rx/machinelearning-examples) repo. For each example, you can bootstrap the python and Bonsai environments from within each example directory. Below, we demonstrate how to get started using the examples by bootstrapping the example environments that are needed to run the example workflow. - -### Windows - -#### Dependencies - -You must have the following installed on your PC in order to run the example workflows: - -- [Python (v3.10)](https://www.python.org/downloads/) *Note: python must be installed using the windows installer and must be added to the system PATH* -- [Git](https://git-scm.com/downloads) -- [Microsoft Visual C++ Redistributable](https://aka.ms/vs/16/release/vc_redist.x64.exe) - -> [!TIP] -> When running windows in a virtual machine, it is necessary to install the [OpenGL mesa drivers](https://github.com/pal1000/mesa-dist-win/releases/) on the windows VM. You can read more about it [here](https://github.com/pal1000/mesa-dist-win) - -#### Instructions - -Open the terminal or powershell. Start by cloning the [Bonsai.ML - Examples](https://github.com/bonsai-rx/machinelearning-examples) repo with: - -```cmd -git clone https://github.com/bonsai-rx/machinelearning-examples.git -``` - -Change directory to one of the examples. For instance, this will bring you to the simulated data example from the LinearDynamicalSystems.Kinematics package: - -```cmd -cd .\machinelearning-examples\examples\LinearDynamicalSystems\Kinematics\SimulatedData -``` - -To create the python virtual environment and install the package, run the following: - -```cmd -python -m venv .venv -.\.venv\Scripts\activate -pip install lds_python@git+https://github.com/joacorapela/lds_python@f761c201f3df883503ecb67acef35ba846e3524c -``` - -> [!TIP] -> If you get an error during this step, you need to make sure that scripts are executable from the terminal. To do this, open a powershell terminal using `Run as Administrator` and enter this line, followed by `yes`: - -```powershell -set-executionpolicy remotesigned -``` - -To create the bonsai environment and install the packages, run the powershell script: - -```cmd -cd .bonsai -powershell .\Setup.ps1 -``` - -Once installed, run the bonsai executable with: - -```cmd -.\Bonsai.exe -``` - -Open the workflow example and start the bonsai workflow. - -### Linux - -#### Notes on Running Bonsai in Linux - -Currently, the examples have only been tested on Ubuntu 22.04. Running Bonsai on Linux is still being tested and should be used with caution. We cannot guarantee that these instructions will work for all Linux distributions or versions. It is important that you consult the general [Bonsai installation guide on Linux](https://github.com/orgs/bonsai-rx/discussions/1101) to ensure that the underlying Bonsai package dependencies are met and installed properly. - -#### Dependencies - -You must have the following installed on your PC in order to run the example workflows: - -- [Python (v3.10)](https://www.python.org/downloads/) *comes installed with the latest version of Ubuntu 22.04* -- [Git](https://git-scm.com/downloads) -- [Mono](https://www.mono-project.com/download/stable/#download-lin) -- [OpenCV and OpenGL binaries](https://github.com/orgs/bonsai-rx/discussions/1101) - - -> [!TIP] -> For installing OpenCV, the simplest method is to follow the instructions on installing OpenCV from pre-built binaries - -#### Instructions - -Open up a terminal and start by cloning the [Bonsai.ML - Examples](https://github.com/bonsai-rx/machinelearning-examples) repo: - -```cmd -git clone https://github.com/bonsai-rx/machinelearning-examples.git -``` - -Change directory to one of the examples. For instance: - -```cmd -cd machinelearning-examples/examples/LinearDynamicalSystems/Kinematics/SimulatedData -``` - -Create the python virtual environment and install the lds python package with the following: - -```cmd -python3 -m venv .venv -source .venv/bin/activate -pip install lds_python@git+https://github.com/joacorapela/lds_python@f761c201f3df883503ecb67acef35ba846e3524c -``` - -> [!TIP] -> If you get an error during this step, it may be because you need to install python virtual environment package on your system. You can do so using the following: - -```cmd -sudo apt install -y python3.10-venv -``` - -Run the `Setup.sh` script using the following: - -```cmd -cd .bonsai -./Setup.sh -``` - -> [!TIP] -> The `Setup.sh` script uses the `xmllint` and `xmlstarlet` packages to convert the assembly location paths in the bonsai config file from Windows to Linux paths. If you do not have these packages already installed on your computer, you can install the `xmllint` and the `xmlstarlet` packages using: - -```cmd -sudo apt install -y libxml2-utils xmlstarlet -``` - -Once the bonsai environment has been created, activate the bonsai environment and run the Bonsai executable: - -```cmd -source ./activate -bonsai -``` - -> [!TIP] -> If your desktop theme is set to dark mode, the Bonsai GUI can display in weird ways due to the way that mono tries to use the desktop theme in applications. If you use the `bonsai-clean` command instead of the `bonsai` command, this will reset the theme that mono uses to the default theme, which can imporove the appearance of the Bonsai GUI. - -Open the workflow example and start the bonsai workflow. diff --git a/examples/Torch/NeuralNetsTorchHub/README.md b/examples/Torch/NeuralNetsTorchHub/README.md index a2636a5..1100e1a 100644 --- a/examples/Torch/NeuralNetsTorchHub/README.md +++ b/examples/Torch/NeuralNetsTorchHub/README.md @@ -4,16 +4,16 @@ In the following example, you will see how to run inference using a model downlo ### Instructions -This example uses the [uv manager](https://docs.astral.sh/uv/) to manage package dependencies and boostrap the environment. To install `uv`, follow the instructions [here](https://docs.astral.sh/uv/getting-started/installation/). +This example uses the [uv manager](https://docs.astral.sh/uv/) to manage package dependencies and bootstrap the environment. To install `uv`, follow the instructions [here](https://docs.astral.sh/uv/getting-started/installation/). The folder contains a Python script which uses the [PyTorch](https://pytorch.org) library. The python script, `torchhub_download.py`, is designed to automatically download the `MiDaS` PyTorch model from the web and place it into the correct folder for the Bonsai workflow. You can bootstrap the Python environment and run the script using uv. For this, open a terminal, change to the example directory, and run the script using uv: ```cmd -cd examples/Torch/Neural +cd examples/Torch/NeuralNetsTorchHub uv run torchhub_download.py ``` -Once the script is finished, the model will be saved as `MiDaS_small.pt` inside of the `models` directory. +Once the script is finished, the model will be saved as `MiDaS_small.pt` inside the `models` directory. ### Workflow @@ -24,4 +24,3 @@ Below is the example workflow. ::: The `CameraCapture` node grabs frames from the camera. These frames are then processed and converted into `Tensor` format. The model is loaded using the `LoadScriptModule` node and used as the `Model` property for the `Forward` node, which will perform forward inference of the image using the model. The remaining nodes process and convert the tensor back into an image. -