Skip to content

fix(semantic-layer): classify numeric/temporal field roles by normalized type#486

Open
yustme wants to merge 1 commit into
keboola:mainfrom
yustme:fix/semantic-layer-field-role-classification
Open

fix(semantic-layer): classify numeric/temporal field roles by normalized type#486
yustme wants to merge 1 commit into
keboola:mainfrom
yustme:fix/semantic-layer-field-role-classification

Conversation

@yustme

@yustme yustme commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Problem

kbagent semantic-layer build (and add dataset --deep-fields) classifies every column of a table into a field rolekey, timestamp, measure, or dimension. In practice, numeric measure columns were being classified as dimension, even on fully-typed tables.

Root cause: the heuristic matched raw type names against a Snowflake-flavored whitelist:

_NUMERIC_TYPES = ("NUMBER", "DECIMAL", "FLOAT", "INTEGER", "INT")

Keboola's own basetype for decimals is NUMERIC, which is not in that list. So a column whose Storage basetype is NUMERIC (the common case for revenue/amount/ratio columns on both Snowflake and BigQuery) never matched the numeric test and fell through to dimension. BigQuery native names (INT64, FLOAT64, BIGNUMERIC) weren't recognized either.

Concrete example — a BigQuery metrics table with columns like total_ppc_revenue (NUMERIC), marketplace_gmv (NUMERIC), count_marketplace_orders (INT64): all 50+ numeric columns classified as dimension.

Fix

  • Route the type test through the existing _normalize_field_type closed vocabulary instead of matching raw names, so it is warehouse-agnostic (NUMBER, NUMERIC, INT64, FLOAT64, Keboola basetypes — all resolve correctly).
  • Add the missing BigQuery native names to _FIELD_TYPE_MAP (int64, float64, bignumeric).
  • A DATE/TIMESTAMP-typed column now classifies as timestamp regardless of its name (a bare date column was previously a dimension); the existing name tokens stay as a fallback for untyped columns.

Tests

  • Added regression cases to TestClassifyFieldRole: NUMERIC measure, BigQuery native numeric types, type-driven timestamp, and the untyped-column fallback.
  • Extended the _normalize_field_type parametrization with the BigQuery names.
  • Full tests/test_semantic_layer_service.py suite passes (156 tests); ruff check clean.

Scope / follow-up

This fixes classification when column types are available. A separate follow-up PR addresses the orthogonal issue that alias / linked-bucket tables carry no column datatype metadata locally (the types live on the source table), which leaves basetype empty and is a different code path.

…zed type

The `build`/`add dataset --deep-fields` role heuristic matched raw type
names against a Snowflake-flavored whitelist (`NUMBER`, `DECIMAL`, `FLOAT`,
`INTEGER`, `INT`). Keboola's own basetype for decimals is `NUMERIC`, which
was absent from that list, so numeric measure columns (revenue, amount,
counts, ...) silently fell through to `dimension` on both Snowflake and
BigQuery -- even when the column was correctly typed.

Route the type test through the existing `_normalize_field_type` closed
vocabulary instead, and add the missing BigQuery native names (`INT64`,
`FLOAT64`, `BIGNUMERIC`). Also make a DATE/TIMESTAMP-typed column classify
as `timestamp` regardless of its name (the name tokens stay as a fallback
for untyped columns), so a bare `date` column is no longer a dimension.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant