Skip to content

Refactor measurement dimensions handling in configuration and related… #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tombonfert merged 3 commits into from
May 28, 2026
Merged

Refactor measurement dimensions handling in configuration and related… #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b1cbe6a
Refactor measurement dimensions handling in configuration and related…
tombonfert May 27, 2026
7715afa
formatting
tombonfert May 27, 2026
b2f2b77
Enhance documentation for internal column mappings in configuration
tombonfert May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 29 additions & 25 deletions docs/impulse/docs/config/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,8 @@ Internal column names that mappings can target:
|-----------------|----------------------------------------------------------|
| `container_id` | Container identifier |
| `channel_id` | Channel identifier |
| `tstart`, `tend`| Sample interval start/end (RLE) |
| `tstart`, `tend`| Sample interval start/end on the `channels` table (RLE) |
| `start_ts`, `stop_ts` | Measurement start/stop epoch timestamps on the `container_metrics` table — referenced by `ContainerEvent` to derive event-fact start/end |
| `value` | Sample value (or attribute value on the EAV tag table) |
| `key` | Attribute key on the EAV `container_tags` table |
| `priority` | Tie-breaker column on the `channel_mapping` table |
Expand Down Expand Up @@ -359,39 +360,42 @@ mode-resolution rules and what counts as a definition change.

## measurement_dimensions (optional)

List of `container_metrics` columns to surface into the gold-layer
`measurement_dimension` table.
List of silver-layer `container_metrics` column names to surface into the
gold-layer `measurement_dimension` table. Names pass through unchanged:
whatever you list here is what you get in gold.

**Allowed values:**
Any column present in your silver `container_metrics` table is a valid
entry — there is no closed allow-list. Typical choices include
`container_id`, `uut_id`, `project`, `vehicle_key`, `file_name`,
`file_path`, `start_ts`, `stop_ts`, and `environment`, but any column
your silver schema carries is fair game.

| Value | Description |
|--------------------|----------------------------------------------|
| `container_id` | Container identifier. |
| `uut_id` | Unit-under-test identifier. |
| `project_id` | Project identifier. |
| `vehicle_key` | Vehicle identifier. |
| `file_name` | Source measurement file name. |
| `source_file_path` | Full path to the source file. |
| `start_ts` | Measurement start timestamp. |
| `stop_ts` | Measurement stop timestamp. |
| `environment` | Recording environment (e.g. PUMA, datalogger). |
`container_id` is part of the default list and is recommended for any
real-world config: it is the upsert key used by incremental processing
and the join key between the measurement dimension and the event-fact
tables. If you override `measurement_dimensions` you take full
ownership of what ends up in gold — the framework does not inject
`container_id` for you. Omit it only if you know the consequences for
downstream joins and incremental runs.

**Default:**

```json
[
"container_id",
"uut_id",
"file_name",
"source_file_path",
"start_ts",
"stop_ts",
"project_id",
"environment"
"stop_ts"
]
```

Pick the entries that match the columns actually present in your
`container_metrics_table`. Columns referenced here must exist in your
silver schema; columns that don't appear here are ignored even if they
exist in silver.
If any listed column is not present in the silver `container_metrics`
table when the report runs, the run fails fast with a `ValueError`
naming the missing columns.

**Migration note (pre-0.1):** earlier versions exposed a fixed enum
that renamed two silver columns on the way to gold (`project` →
`project_id`, `file_path` → `source_file_path`). The rename has been
removed; if you previously listed `"project_id"` or `"source_file_path"`,
list `"project"` and `"file_path"` instead. The default list also
shrank — if you relied on the old eight-column default, add the
columns you want explicitly.
151 changes: 133 additions & 18 deletions docs/impulse/docs/data_model/silver_layer_schema.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,26 +104,46 @@ count.
| `duration_ms` | `int` | Yes | Total duration in milliseconds. |
| `num_channels` | `int` | Yes | Number of channels in the container. |

#### Internal columns referenced by the framework

The framework hard-references the following internal names on `container_metrics`. Map any silver column to one of these via
[`solver_config.container_metrics.column_name_mapping`](../config/configuration.md#solver-column-mappings-and-filters)
when your physical column has a different name.

| Internal name | Referenced by |
|---------------|-------------------------------------------------------------------------------|
| `container_id`| Join key against `container_tags`, upsert key in incremental mode |
| `start_ts` | `ContainerEvent` event-fact `start_ts`; default `measurement_dimensions` entry |
| `stop_ts` | `ContainerEvent` event-fact `end_ts`; default `measurement_dimensions` entry |
| `project_id` | `KeyValueStoreSolver` project scoping (when `solver_config.project_id` is set) |

Every other column on `container_metrics` is **pass-through**: it lands in
the gold `measurement_dimension` table verbatim if you list it in
[`measurement_dimensions`](../config/configuration.md#measurement_dimensions-optional),
and can be used in `container_filters.metric_filters`; the framework does
not reference it under any specific internal name.

### Additional columns commonly populated

`container_metrics` typically carries additional metadata columns that
get surfaced into the gold-layer `measurement_dimension` table when
listed in the report's
[`measurement_dimensions`](../config/configuration.md#measurement_dimensions-optional)
config. The framework recognises the following names through the
`MeasurementDimensions` enum — populate any subset that fits your
data; none are required by the engine.

| Column | Type | Description |
|--------------------|----------|----------------------------------------------|
| `uut_id` | `long` | Unit-under-test identifier. |
| `vehicle_key` | `string` | Vehicle identifier. |
| `project_id` | `long` | Project identifier. |
| `file_name` | `string` | Source measurement file name. |
| `source_file_path` | `string` | Full path to the source file. |
| `start_ts` | `long` | Measurement start timestamp (epoch). |
| `stop_ts` | `long` | Measurement stop timestamp (epoch). |
| `environment` | `string` | Recording environment (e.g. PUMA, datalogger). |
config. Any column you list there must exist in this table — names
pass through to gold unchanged. The columns below are common choices,
but `measurement_dimensions` accepts any column you carry on this
table.

| Column | Type | Description |
|---------------|----------|----------------------------------------------|
| `uut_id` | `long` | Unit-under-test identifier. |
| `vehicle_key` | `string` | Vehicle identifier. |
| `project` | `long` | Project identifier. |
| `file_name` | `string` | Source measurement file name. |
| `file_path` | `string` | Full path to the source file. |
| `start_ts` | `long` | Measurement start timestamp (epoch). |
| `stop_ts` | `long` | Measurement stop timestamp (epoch). |
| `environment` | `string` | Recording environment (e.g. PUMA, datalogger). |

:::note Two timestamp conventions

Expand All @@ -132,10 +152,9 @@ section) and `start_ts`/`stop_ts` (epoch long, listed here) are
**different columns**, not naming variants. Real-world
`container_metrics` tables typically carry both: `start_dt`/`stop_dt`
for human-readable display, `start_ts`/`stop_ts` for the gold
`measurement_dimension` because the corresponding
`MeasurementDimensions` enum values map to the epoch-typed columns.
Populate whichever your queries and `measurement_dimensions` config
need.
`measurement_dimension` (the default `measurement_dimensions` includes
the epoch-typed pair). Populate whichever your queries and
`measurement_dimensions` config need.

:::

Expand All @@ -154,6 +173,20 @@ TSAL queries select recordings by tag key (e.g.
| `key` | `string` | Yes | Tag key (e.g. `"vehicle_key"`, `"project_id"`). |
| `value` | `string` | Yes | Tag value. |

#### Internal columns referenced by the framework

Map any silver column to these via
[`solver_config.container_tags.column_name_mapping`](../config/configuration.md#solver-column-mappings-and-filters)
when your physical column has a different name.

| Internal name | Referenced by |
|---------------|------------------------------------------------------------------------------------------------|
| `container_id`| Join key into `container_metrics` after tag filtering |
| `key` | EAV pivot key — driven by `query.havingTag(...)` and `container_filters.tag_filters` |
| `value` | EAV pivot value — driven by `query.havingTag(...)` and `container_filters.tag_filters` |
| `project_id` | `KeyValueStoreSolver` project scoping (when `solver_config.project_id` is set, and this table carries a project column) |
| `parent_id` | Optional per-table filter target (e.g. `solver_config.container_tags.filters = {"parent_id": ...}`) |

---

## channel_metrics
Expand Down Expand Up @@ -191,6 +224,20 @@ via `COALESCE(channel_metrics.unit, channel_mapping.source_unit)`. The
column is not part of the canonical schema — omit it for layouts that
don't need per-channel physical units.

#### Internal columns referenced by the framework

Map any silver column to these via
[`solver_config.channel_metrics.column_name_mapping`](../config/configuration.md#solver-column-mappings-and-filters)
when your physical column has a different name.

| Internal name | Referenced by |
|----------------|--------------------------------------------------------------------------------------------|
| `container_id` | Composite join key with `channels` and `channel_tags` |
| `channel_id` | Composite join key with `channels` and `channel_tags` |
| `channel_name` | Default join target on the `channel_mapping`→`channel_metrics` alias-resolution join |
| `data_key` | Default join target on the `channel_mapping`→`channel_metrics` alias-resolution join |
| `unit` | Authoritative source unit on aliased reads (takes precedence over `channel_mapping.source_unit`) |

---

## channel_tags
Expand All @@ -207,6 +254,20 @@ selectors look up signals by tag key (e.g.
| `key` | `string` | Yes | Tag key (e.g. `"channel_name"`, `"brand"`, `"model"`). |
| `value` | `string` | Yes | Tag value. |

#### Internal columns referenced by the framework

Map any silver column to these via
[`solver_config.channel_tags.column_name_mapping`](../config/configuration.md#solver-column-mappings-and-filters)
when your physical column has a different name. Note: `channel_tags` is
read by `DeltaSolver` only.

| Internal name | Referenced by |
|---------------|------------------------------------------------------------------------------|
| `container_id`| Composite join key with `channel_metrics` and `channels` |
| `channel_id` | Composite join key with `channel_metrics` and `channels` |
| `key` | EAV pivot key — driven by `query.channel(...)` selector kwargs |
| `value` | EAV pivot value — driven by `query.channel(...)` selector kwargs |

---

## channels
Expand Down Expand Up @@ -249,6 +310,24 @@ constructed with `drop_implausible_data=True` — in that mode, samples
with `is_plausible = False` are filtered before RLE encoding. If the
flag is `False` (the default), the column is ignored and may be omitted.

#### Internal columns referenced by the framework

Map any silver column to these via
[`solver_config.channels.column_name_mapping`](../config/configuration.md#solver-column-mappings-and-filters)
when your physical column has a different name.

| Internal name | Referenced by |
|---------------|----------------------------------------------------------------------------------------|
| `container_id`| Composite key joining samples back to their container |
| `channel_id` | Composite key joining samples back to their channel |
| `tstart` | Sample interval start (RLE format) — consumed by the solve UDF |
| `tend` | Sample interval end (RLE format) — consumed by the solve UDF |
| `value` | Sample value — consumed by the solve UDF and aggregations |

For raw-format `channels`, the same internal names apply except that
`timestamp` replaces the `tstart`/`tend` pair; the engine derives `tend`
during raw→RLE conversion.

---

## channel_mapping (optional)
Expand Down Expand Up @@ -287,6 +366,29 @@ conversions to the same channel in the same query. Workarounds: select
the conflicting aliases in **separate queries**, or align the mapping rows
so they agree on the unit pair per physical channel.

#### Internal columns referenced by the framework

Map any silver column to these via
[`solver_config.channel_mapping.column_name_mapping`](../config/configuration.md#solver-column-mappings-and-filters)
when your physical column has a different name. Note: `channel_mapping`
is read by `KeyValueStoreSolver` only.

| Internal name | Referenced by |
|------------------|------------------------------------------------------------------------------------------------|
| `channel_alias` | The user-facing alias selector kwarg in `query.channel_with_alias(channel_alias=...)` |
| `channel_name` | Default join target on the `channel_mapping`→`channel_metrics` join |
| `data_key` | Default join target on the `channel_mapping`→`channel_metrics` join |
| `source_channel` | Alias-resolution source on the `channel_mapping`→`channel_metrics` join |
| `priority` | Tie-breaker when multiple physical channels match a logical alias |
| `project_id` | `KeyValueStoreSolver` project scoping (when `solver_config.project_id` is set) |
| `source_unit` | Source unit fallback (when paired with a `unit_conversion_table`) |
| `target_unit` | Target unit for aliased reads (when paired with a `unit_conversion_table`) |

To override the default `(source_channel, channel_name) + (data_key, data_key)`
composite join, set
[`channel_mapping.join_keys`](../config/configuration.md#alias-resolution-join-keys-optional)
explicitly.

---

## unit_conversion (optional)
Expand All @@ -307,5 +409,18 @@ whose `unit` matches `source_unit`) and `target_factor` (the row whose
multiplies values by `source_factor / target_factor`. Missing rows or a
`group_id` mismatch yield a null factor and no conversion.

#### Internal columns referenced by the framework

Map any silver column to these via
[`solver_config.unit_conversion.column_name_mapping`](../config/configuration.md#solver-column-mappings-and-filters)
when your physical column has a different name. Note: `unit_conversion`
is read by `KeyValueStoreSolver` only.

| Internal name | Referenced by |
|----------------------|--------------------------------------------------------------------------------------------|
| `group_id` | Constraint that source and target units live in the same family |
| `unit` | Lookup key matched against `channel_mapping.source_unit` / `target_unit` |
| `conversion_factor` | Multiplier to the family's base unit; combined into the per-channel `source/target` factor |

Configured via `source.unit_conversion_table` (see
[Configuration](../config/configuration.md)).
10 changes: 10 additions & 0 deletions src/impulse_query_engine/analyze/query/solvers/solver_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,16 @@ def tend_col(self) -> str:
"""Internal column name for the end timestamp."""
return "tend"

@property
def start_ts_col(self) -> str:
"""Internal column name for the measurement-start epoch timestamp on container_metrics."""
return "start_ts"

@property
def stop_ts_col(self) -> str:
"""Internal column name for the measurement-stop epoch timestamp on container_metrics."""
return "stop_ts"

@property
def value_col(self) -> str:
"""Internal column name for the signal value on the channels table."""
Expand Down
Loading
Loading