4.0 blog

Author: Billy Charlton

docs/CHANGELOG-header.md

@@ -10,3 +10,4 @@ The list of changes is generated automatically based on the commit messages them
 - See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
 - The SimWrapper code repository containing the full commit history is on GitHub at <https://github.com/simwrapper/simwrapper>.
+

docs/CHANGELOG.md

@@ -10,11 +10,12 @@ The list of changes is generated automatically based on the commit messages them
 - See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
 - The SimWrapper code repository containing the full commit history is on GitHub at <https://github.com/simwrapper/simwrapper>.
+
 # Changelog
 
 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
-## 4.0.0 (2025-06-10)
+## 4.0.0 (2025-06-11)
 
 Bump to version 4.0. New front page, file view, color scheme, event viewer (beta), map builder
 (beta), and updated Python (pip) package for running standalone or in server environments.

docs/assets/default-dashboard.png

@@ -1,6 +1,6 @@
 ---
 id: guide-dashboards
-title: 2. Dashboards in depth
+title: 3. Dashboards in depth
 ---
 
 A dashboard is a page laid out with multiple charts, plots, and visualizations all together. You define the layout with a YAML configuration file which contains the types of plots and their configurations all in one place.

docs/assets/example-folder.jpg

@@ -3,42 +3,43 @@ id: guide-getting-started
 title: 1. Getting started tutorial
 ---
 
-Welcome to SimWrapper! Let's get you up and running with the basics.
+Welcome to SimWrapper! Let's get you up and running with the basics. This guide uses sample data that's hopefully a lot like the data you have for your projects
 
-- This guide uses sample data that's hopefully a lot like the data you would have for your projects
-- Use Google Chrome or MS Edge for the guide. Other browsers (Firefox, Safari) require a [separate local HTTP server](file-management) to access local files; for now that's just a stumbling block to getting started.
+> ➡ Use a Chromium-based browser: **Chrome/Edge/Brave** and others, for this guide.<br/><br/>Other browsers (Firefox, Safari) can work, but require a [separate local HTTP server](file-management) to access local files on your computer. For now, that's just a stumbling block to getting started. Safari is a particularly bad browser for data visualization, since Apple doesn't support the latest APIs nor local file access, and Safari has the slowest Javascript engine! 🐳
 
 ## How it works: SimWrapper and file-based configuration
 
-Most MATSim/ActivitySim outputs such as the `*.xml.gz` files are too large to open in a web browser, so SimWrapper provides a set of _visualization plugins_ which can display something useful for you. Plugins exist for lots of things and the list is growing: link volumes, agent animations, aggregate area summaries, and more.
+Your transport simulation produces a lot of files for every run, some of them quite large! MATSim/ActivitySim outputs such as the `*.xml.gz` files are usually too big to open in a web browser. SimWrapper provides a set of _visualization plugins_ which load the raw data, or more often, load post-processed files which contain only the data that you wish to display. Plugins exist for lots of things and the list is growing: link volumes, agent animations, aggregate area summaries, and more.
 
-Here's how it works: For every visualization you want to create, you write a small _configuration file_ and store it in the same folder as the inputs for that visualization. We use the YAML text format, which is a common configuration file format. For each properly named YAML file, one visualization thumbnail will appear in that folder when you navigate to the folder in SimWrapper. Clicking on the thumbnail will open that visualization full-screen.
+On top of this, you can collect various visualizations into _dashboards_ that you can lay out logically to help your users find the data they need.
 
-Generally, a viz will require a specific set of inputs, and those inputs are usually the result of some _post-processing_ of the raw simulation outputs. It's up to you to do that post-processing and store the files in the same folder as your config file.
+Here's how it works: For every visualization or dashboard you want to create, you write a small _configuration file_ and store it in the same folder as the inputs for that visualization. These configs are in the simple [YAML text format](https://yaml.org/) and SimWrapper looks for YAML files with specific names to learn what it should read and display (more on this below).
+
+Generally, a viz will require a specific set of inputs, and those inputs are either raw simulation outputs or the result of some _post-processing_ of the outputs. It's up to you to do that post-processing and store the resulting files in the same folder as your YAML config files.
 
 Let's get started with some sample data.
 
 ## 1. Get the sample data and open it in SimWrapper
 
 - Download [simwrapper-example-project.zip](https://github.com/simwrapper/simwrapper-example-project/archive/refs/heads/main.zip) from GitHub
 - Unzip the file somewhere you can find it easily - Desktop, home folder, etc.
-- Go to [simwrapper.github.io](https://simwrapper.github.io/site) and click `Add folder...` and browse to the folder you just created. Grant access to the folder so the SimWrapper site can see the files!
-  - (If you are using Firefox, `cd` to the data folder and run `simwrapper here` to start the local HTTP server)
+- Go to https://simwrapper.app and click `View local files...` and browse to the folder you just created. Grant access to the folder so the SimWrapper website can see the files!
+- Click on the `data` folder. You will find dashboard examples and some raw data from a few MATSim runs.
+
+If you are using Firefox, there is no "View local files" button. Instead `cd` to the data folder and run `simwrapper here` to start the local HTTP server. This requires first running `pip install simwrapper` to get the simwrapper  command-line helper app. I suggest you use Chrome/Edge/Brave for this tutorial!
 
 You should now see something similar to this:
 
 <img src="assets/example-folder.jpg" style="border: 1px solid #ccc"><i>Example data folder</i></img>
 
-## 2. Explore the samples
-
 Each of the subfolders in the example project shows different map views and capabilities of SimWrapper -- network link plots, statistical charts, area maps (shapefiles), dashboards, and so on.
 
-- Experiment with the various knobs and configuration settings to see how the visualizations can be manipulated
-- From your PC file browser, open up the `viz-*.yaml` files in each subfolder to see how each of the visualizations is defined in a readable text format.
+- Experiment with the various knobs and configuration settings in the visualizations to see how they can be manipulated
+- From your PC file explorer / Finder, find the YAML text files named`viz-*.yaml` in each subfolder to see how each of the visualizations is defined in a clear, readable text format.
 - Every visualization type has a different filename "prefix" to help you find them: e.g, `viz-map-*.yaml` are for shapefiles, `viz-link-*.yaml` are for MATSim network plots, and so on.
-- You can edit these YAML files, save, and click Reload on your browser to see how your changes affect the visualizations.
+- Try making a copy and editing these YAML files. Save and click `Reload` on your browser to see how your changes affect the visualizations.
 
-## 3. Create a dashboard with some charts
+## 2. Create a dashboard with some charts
 
 The dashboards subfolder shows how you can combine multiple visualizations into cohesive dashboards.
 
@@ -49,17 +50,29 @@ See the [Dashboards in Depth](guide-dashboards) article to learn more about buil
 
 ## 4. Configuring dashboard templates for multiple run folders
 
-In SimWrapper, everything is folder-based. So `viz-*.yaml` and `dashboard-*.yaml` files in a folder will automatically be detected and loaded based on their filenames.
+In SimWrapper, everything is file- and folder-based. So `viz-*.yaml` and `dashboard-*.yaml` files in a folder will automatically be detected and loaded, based on their filenames.
 
 If you want to define dashboards that will be used for **multiple folders**, such as several runs for a particular project:
 
-- **Create a folder** named `simwrapper` in the parent project directory.
+- **Create a folder** named `simwrapper` or `.simwrapper` in the parent project directory.
 - Move all dashboard, viz, and template YAMLs into that folder
 - Tweak any file paths as necessary, so that relative file names resolve properly.
 - The base folder for a dashboard is the _folder you are viewing_, not the dashboard template folder.
-- You can have multiple `simwrapper` folders all the way up your folder hierarchy; dashboard panels will be generated based on filename, and each found dashboard will be displayed as a tab on the folder view.
+- You can have multiple `simwrapper` folders all the way up your folder hierarchy; dashboard panels will be generated based on filename, and each found dashboard will be displayed as a tab on the dashboard.
+
+## 5. More on YAML filenames
 
-## 5. More details on visualizations and their YAML files
+SimWrapper looks in your folder for YAML files that follow specific naming conventions. That way, it won't try to read unrelated YAML files that you might have. But this means you must name your configuration files carefully! SimWrapper will **silently ignore all YAML files** that do not match these conventions:
+
+- **`dashboard-*.yaml`** - these contain the configuration settings for dashboard panes. Each found file results in a left-side tab on the dashboard page, and that tab contains the dashboard contents specified in this file. For this reason, many users tend to number their dashboard configs so that the tabs always appear in a predictable order: `dashboard-1-summary.yaml`, `dashboard-2-mode-choice.yaml`, `dashboard-3-traffic.yaml` and so on.
+- **`viz-*.yaml`** - these contain individual map-based visualizations. Each of these will appear on the Files tab of a folder.
+  - Each visualization plugin has a specific subname that must follow `viz-subname-...` in the filename: for example, the shapefile viewer would be `viz-map-*.yaml` while the xy-hexagon viewer is `viz-xy-*.yaml`. See the API reference for each plugin for examples.
+  - Use these `viz-*` configs if you don't have the need to collate multiple vizes into dashboards, or if you want single visualizations to have their own URL that you can link to directly.
+- **`simwrapper-config.yaml`** - is a special top-level configuration for project sites that is described in detail on [Building a project site](guide-project-sites).
+
+Note that in most cases, both `.yaml` and `.yml` file extensions are valid and honored.
+
+## 6. Example roadway links visualization
 
 Here is an example YAML config file for a link-volume summary:
 
@@ -80,8 +93,6 @@ If you wanted to look at several different link volume plots from the same model
 
 This is a very different paradigm than most "point and click" GIS tools, but we have found that the ability to script and cut/paste the config files has been a huge time saver and also reduces manual errors.
 
-> Make sure that your files are allowed to be "world-readable" before you publish anything to public-svn! Once files are pushed to public-svn, they are not secured in any way; anyone on the internet can access them!
-
 ### Visualization types
 
 Each visualization is described in the API Reference section of this documentation, including how to post-process your outputs if necessary, and how to define any configuration settings for your visualizations.

docs/guide-dashboards.md

@@ -0,0 +1,198 @@
+---
+id: guide-dashboards-from-code
+title: 2. MATSim dashboards from code
+---
+
+MATSim users should take advantage of the [MATSim SimWrapper Contrib](https://github.com/matsim-org/matsim-libs/tree/main/contribs/simwrapper) which makes building SimWrapper dashboards very easy.
+
+Including the simwrapper contrib allows you to write Java code directly in your simulation that builds dashboards for you -- in most cases you don't need to write any YAML at all.
+
+By adding a single line to your MATSim project config, you will get a _default SimWrapper dashboard_ that gets built automatically when you run your simulation. From there, you can add further project-specific dashboards as needed.
+
+## Including the SimWrapper contrib
+
+First, make sure the simwrapper contrib is included and activated in your project `pom.xml`:
+
+```xml
+<dependencies>
+  <!-- ... -->
+    <dependency>
+			<groupId>org.matsim.contrib</groupId>
+			<artifactId>simwrapper</artifactId>
+			<version>${matsim.version}</version>
+    </dependency>
+</dependencies>
+```
+
+Then, in your run class, add this one line wherever you add your other project modules:
+
+```java
+controler.addOverridingModule(new SimWrapperModule())
+```
+
+That's it. When you run your simulation, the contrib will automatically generate the necessary `dashboard-*.yaml` files in your output folder.<br/>Open [simwrapper.app](https://simwrapper.app), open your local output folder, and the dashboards should be there.
+
+![default dashboard](assets/default-dashboard.png)
+
+## Creating your own dashboards from code
+
+Now that you have the default dashboard working, you can start adding your own dashboards.
+
+There are fully-executed examples in the following open scenarios:
+
+- https://github.com/matsim-scenarios/matsim-berlin
+- https://github.com/matsim-scenarios/matsim-leipzig
+
+Explore the files in `src/main/java/org/matsim/dashboard` for examples on creating post-processed CSV outputs and the dashboard configurations.
+- In particular this example dashboard is very feature-rich: https://github.com/matsim-scenarios/matsim-leipzig/blob/main/src/main/java/org/matsim/dashboard/CycleHighwayDashboard.java
+
+### IntelliJ autocomplete
+
+While writing your dashboard code, take advantage of IntelliJ's autocomplete feature. At each step of writing the code below, pressing `Tab` after any of the dots will drop down an autocomplete with all of the YAML parameters available on that object. This is far less error-prone than trying to decipher the YAML configuration options from the SimWrapper documentation itself.
+
+The SimWrapper docs are the primary source of truth, however. If you find discrepancies between what IntelliJ thinks and what the SimWrapper docs say, try both and report back to us if there are problems!
+
+**Example dashboard-building Java code:**
+```java
+Config config = ConfigUtils.createConfig();
+
+SimWrapper sw = SimWrapper.create(config);
+
+SimWrapperConfigGroup simwrapperCfg = ConfigUtils.addOrGetModule(config, SimWrapperConfigGroup.class);
+simwrapperCfg.defaultDashboards = SimWrapperConfigGroup.Mode.disabled;
+simwrapperCfg.sampleSize = 0.25;
+simwrapperCfg.defaultParams().mapCenter = "12.38,51.34";
+simwrapperCfg.defaultParams().mapZoomLevel = 6.8;
+
+//skip default dashboards
+simwrapperCfg.defaultDashboards = SimWrapperConfigGroup.Mode.disabled;
+
+//add dashboards according to command line parameters
+if (cycleHighwayAnalysis == CycleHighwayAnalysis.ENABLED) {
+  sw.addDashboard(Dashboard.customize(new CycleHighwayDashboard(baseDir, shp.getShapeFile(), highwaysShpPath)).context("cycle-highway"));
+}
+
+try {
+  sw.generate(runDirectory);
+  sw.run(runDirectory);
+} catch (IOException e) {
+  throw new InterruptedIOException();
+}
+```
+
+**Snippet from CycleHighwayDashboard.java:**
+```java
+package org.matsim.dashboard;
+
+import org.matsim.analysis.CycleHighwayAnalysis;
+import org.matsim.application.analysis.traffic.TrafficAnalysis;
+import org.matsim.application.prepare.network.CreateGeoJsonNetwork;
+import org.matsim.simwrapper.Dashboard;
+import org.matsim.simwrapper.Header;
+import org.matsim.simwrapper.Layout;
+import org.matsim.simwrapper.viz.*;
+import tech.tablesaw.plotly.traces.BarTrace;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Shows information about an optional policy case, which implements cycle highways in Leipzig.
+ * It also compares the agents and their trips using the cycle highways with their respective trips in the base case.
+ */
+public class CycleHighwayDashboard implements Dashboard {
+	private final String basePath;
+	private final String shp;
+	private final String highwaysShp;
+
+	CycleHighwayDashboard(String basePath, String shp, String highwaysShp) {
+		if (!basePath.endsWith("/")) 			basePath += "/";
+		this.basePath = basePath;
+		this.shp = shp;
+		this.highwaysShp = highwaysShp;
+	}
+
+	@Override
+	public void configure(Header header, Layout layout) {
+		header.title = "Cycle Highways Dashboard";
+		header.description = "Shows statistics about agents, who used the newly implemented cycle highway " +
+			"and compares to the corresponding trips in the base case.";
+
+		String[] args = new ArrayList<>(List.of("--base-path", basePath, "--shp", shp, "--highways-shp-path", highwaysShp)).toArray(new String[0]);
+
+		layout.row("first")
+			.el(Tile.class, (viz, data) -> {
+				viz.dataset = data.compute(CycleHighwayAnalysis.class, "mean_travel_stats.csv", args);
+				viz.height = 0.1;
+			});
+
+		layout.row("modalSplit")
+			.el(Plotly.class, (viz, data) -> {
+				viz.title = "Modal split";
+
+				viz.layout = tech.tablesaw.plotly.components.Layout.builder()
+					.barMode(tech.tablesaw.plotly.components.Layout.BarMode.STACK)
+					.build();
+
+				Plotly.DataSet ds = viz.addDataset(data.compute(CycleHighwayAnalysis.class, "mode_share.csv", args))
+					.constant(SOURCE, "Policy")
+					.aggregate(List.of(MAIN_MODE), SHARE, Plotly.AggrFunc.SUM);
+
+				Plotly.DataSet dsBase = viz.addDataset(data.compute(CycleHighwayAnalysis.class, "mode_share_base.csv", args))
+					.constant(SOURCE, "Base")
+					.aggregate(List.of(MAIN_MODE), SHARE, Plotly.AggrFunc.SUM);
+
+				viz.mergeDatasets = true;
+
+				viz.addTrace(BarTrace.builder(Plotly.OBJ_INPUT, Plotly.INPUT).orientation(BarTrace.Orientation.HORIZONTAL).build(),
+					ds.mapping()
+						.name(MAIN_MODE)
+						.y(SOURCE)
+						.x(SHARE)
+				);
+
+		layout.row("locations")
+			.el(Hexagons.class, (viz, data) -> {
+
+				viz.title = "Cycle highway trips - Origins";
+				viz.center = data.context().getCenter();
+				viz.zoom = data.context().mapZoomLevel;
+				viz.height = 7.5;
+				viz.width = 2.0;
+				viz.file = data.compute(CycleHighwayAnalysis.class, "cycle_highway_agents_trip_start_end.csv");
+				viz.projection = CRS;
+				viz.addAggregation("trip origins", "person", "start_x", "start_y");
+			})
+			.el(Hexagons.class, (viz, data) -> {
+				viz.title = "Cycle highway trips - Destinations";
+				viz.center = data.context().getCenter();
+				viz.zoom = data.context().mapZoomLevel;
+				viz.height = 7.5;
+				viz.width = 2.0;
+				viz.file = data.compute(CycleHighwayAnalysis.class, "cycle_highway_agents_trip_start_end.csv");
+				viz.projection = CRS;
+				viz.addAggregation("trip destinations", "person", "end_x", "end_y");
+			})
+
+		layout.row("volumes")
+			.el(MapPlot.class, (viz, data) -> {
+				viz.title = "Simulated traffic volume by bike";
+				viz.center = data.context().getCenter();
+				viz.zoom = data.context().mapZoomLevel;
+				viz.height = 7.5;
+				viz.width = 2.0;
+				viz.setShape(data.compute(CreateGeoJsonNetwork.class, "network.geojson", "--with-properties", "--mode-filter", "car,freight,drt,bike"), "id");
+				viz.addDataset(TRAFFIC, data.compute(TrafficAnalysis.class, "traffic_stats_by_link_daily.csv", "--transport-modes" , "car,bike,freight"));
+
+				viz.display.lineColor.dataset = TRAFFIC;
+				viz.display.lineColor.columnName = "vol_bike";
+				viz.display.lineColor.join = "link_id";
+				viz.display.lineColor.setColorRamp(ColorScheme.RdYlBu, 5, true);
+				viz.display.lineWidth.dataset = TRAFFIC;
+				viz.display.lineWidth.columnName = "vol_bike";
+				viz.display.lineWidth.scaleFactor = 20000d;
+				viz.display.lineWidth.join = "link_id";
+			});
+	}
+}
+```