halld-ai

Great Expectations (GE) validation of Jefferson Lab Hall D (GlueX) data: physics flat trees, RCDB run conditions and CCDB calibration constants, with every result in one Data Docs site and a Grafana trend dashboard.

Setup

uv sync

The flat-tree example needs the public reference tree:

curl -sL -o data/gluex/flat_data.root \
  https://raw.githubusercontent.com/JeffersonLab/gluex_workshops/master/tutorial_2019/session0h/MakeFlatTrees/flat_data.root

The database examples need network access to hallddb.jlab.org. CCDB has no PyPI release: clone https://github.com/JeffersonLab/ccdb and put its python/ directory on PYTHONPATH, or set CCDB_HOME to the checkout.

Examples

Run from the repo root, for example uv run examples/01_flat_tree.py.

Example	Validates
01_flat_tree.py	GlueX flat ROOT tree, one row per particle combo
02_rcdb_last_runs.py	run conditions of the 20 newest runs
03_rcdb_run_period.py	run conditions of a whole run period
04_ccdb_constants.py	calibration constants for a run range, two table shapes
05_beam_energy_consistency.py	RCDB beam energy (MeV) against CCDB endpoint energy (GeV)

Each example fetches a DataFrame (sources/), builds a suite (expectations/) and calls halld_ge.validate.validate(). Adding a check is one line in an expectations/ module; adding a data kind is one file each in sources/, expectations/ and examples/.

Results

• Terminal: pass/fail per suite, one line per failed expectation.
• results/<run_name>.json: the raw validation result.
• gx/uncommitted/data_docs/local_site/index.html: all validations across all sources in one sortable index, plus a contract page per suite. Publish with uv run halld_ge/publish_docs.py <dir>; bound the history with uv run halld_ge/prune_validations.py.
• results/validations.sqlite: one row per expectation per run. sh grafana/run_grafana.sh starts a provisioned Grafana on http://localhost:3000 with pass-rate and drift panels over it.

Documentation

• docs/design.md — engine choice, row grain per data kind, what GE does not cover.
• docs/operations.md — connections, run modes, where thresholds come from, hosting and pruning.