fix(semantic-layer): classify numeric/temporal field roles by normalized type#486
Open
yustme wants to merge 1 commit into
Open
fix(semantic-layer): classify numeric/temporal field roles by normalized type#486yustme wants to merge 1 commit into
yustme wants to merge 1 commit into
Conversation
…zed type The `build`/`add dataset --deep-fields` role heuristic matched raw type names against a Snowflake-flavored whitelist (`NUMBER`, `DECIMAL`, `FLOAT`, `INTEGER`, `INT`). Keboola's own basetype for decimals is `NUMERIC`, which was absent from that list, so numeric measure columns (revenue, amount, counts, ...) silently fell through to `dimension` on both Snowflake and BigQuery -- even when the column was correctly typed. Route the type test through the existing `_normalize_field_type` closed vocabulary instead, and add the missing BigQuery native names (`INT64`, `FLOAT64`, `BIGNUMERIC`). Also make a DATE/TIMESTAMP-typed column classify as `timestamp` regardless of its name (the name tokens stay as a fallback for untyped columns), so a bare `date` column is no longer a dimension.
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
kbagent semantic-layer build(andadd dataset --deep-fields) classifies every column of a table into a field role —key,timestamp,measure, ordimension. In practice, numeric measure columns were being classified asdimension, even on fully-typed tables.Root cause: the heuristic matched raw type names against a Snowflake-flavored whitelist:
Keboola's own basetype for decimals is
NUMERIC, which is not in that list. So a column whose Storage basetype isNUMERIC(the common case for revenue/amount/ratio columns on both Snowflake and BigQuery) never matched the numeric test and fell through todimension. BigQuery native names (INT64,FLOAT64,BIGNUMERIC) weren't recognized either.Concrete example — a BigQuery metrics table with columns like
total_ppc_revenue(NUMERIC),marketplace_gmv(NUMERIC),count_marketplace_orders(INT64): all 50+ numeric columns classified asdimension.Fix
_normalize_field_typeclosed vocabulary instead of matching raw names, so it is warehouse-agnostic (NUMBER,NUMERIC,INT64,FLOAT64, Keboola basetypes — all resolve correctly)._FIELD_TYPE_MAP(int64,float64,bignumeric).DATE/TIMESTAMP-typed column now classifies astimestampregardless of its name (a baredatecolumn was previously adimension); the existing name tokens stay as a fallback for untyped columns.Tests
TestClassifyFieldRole: NUMERIC measure, BigQuery native numeric types, type-driven timestamp, and the untyped-column fallback._normalize_field_typeparametrization with the BigQuery names.tests/test_semantic_layer_service.pysuite passes (156 tests);ruff checkclean.Scope / follow-up
This fixes classification when column types are available. A separate follow-up PR addresses the orthogonal issue that alias / linked-bucket tables carry no column datatype metadata locally (the types live on the source table), which leaves
basetypeempty and is a different code path.