feat(semantic-layer): resolve alias/linked-bucket column types (workspace, auto by default in UI)#487
Open
yustme wants to merge 3 commits into
Open
Conversation
…rkspace Alias and linked-bucket tables carry no per-column datatype metadata in Storage -- the types live on the source table, so `columnMetadata` on the alias is empty (verified against both BigQuery and Snowflake; it is an alias trait, not a backend one). The build heuristic therefore saw empty basetypes and classified every field as `dimension`. Add an opt-in `kbagent semantic-layer build --types-workspace <ID>`: for tables whose Storage metadata carries no column types, query the warehouse `INFORMATION_SCHEMA.COLUMNS` through the given workspace (Query Service) and backfill the real types before generation. Backend-aware SQL for BigQuery and Snowflake; identifiers are validated (not escaped) against a strict charset before interpolation. Unresolved tables are reported non-fatally in the build result as `type_resolution_errors`. Without the flag, behaviour is unchanged. Complements the field-role classifier fix (keboola#486): this makes the types available, that maps a numeric measure column to the `measure` role.
… type resolution The workspace type-resolution path keys the INFORMATION_SCHEMA dialect off the table's storage backend, but get_table_detail did not expose it, so --types-workspace failed with "unsupported backend ''" and every field still defaulted to dimension. Add `backend` (from the owning bucket) to the get_table_detail response and cover it with a test.
Wire type resolution through the server /build endpoint and make it work from the UI without any extra input: BuildRequest gains `types_workspace_id` (explicit) and `auto_resolve_types` (default True), and the endpoint passes both to build_model. In auto mode the service picks a Query-Service-capable, read-only workspace whose backend matches each table (via list_workspaces), cached per backend. The enrichment helper now takes a `resolve_workspace_id(backend)` callable instead of a fixed id, so the explicit-id and auto paths share one code path. No workspace found -> reported non-fatally, build still proceeds. CLI keeps explicit `--types-workspace`; adds `--auto-types-workspace` for parity. Because the server default is auto, an unchanged UI build now resolves alias/linked-bucket column types instead of defaulting every field to dimension.
yustme
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
kbagent semantic-layer buildclassified every field of alias / linked-bucket tables asdimension— including numeric measures.Root cause (verified against live projects): an alias table carries no per-column datatype metadata in Storage.
isTypedistrue, butcolumnMetadataanddefinition.columnsare both empty — the types live on the source table, which for a cross-project linked bucket isn't reachable with the consuming project's token. So the build heuristic receives empty basetypes and defaults everything todimension.This is an alias trait, not a backend one — confirmed on both:
columnMetadatapopulatedcolumnMetadatapopulatedFix
Resolve the real column types from the warehouse
INFORMATION_SCHEMA(via the Query Serviceexecute_query) for any table whose Storage metadata carries none, then backfill them before generation.Two ways to supply the workspace:
build ... --types-workspace <ID>.--auto-types-workspace): the service picks a Query-Service-capable, read-only workspace whose backend matches each table, vialist_workspaces(cached per backend).The server
/buildendpoint gainstypes_workspace_idandauto_resolve_types(default True), so an unchanged UI build now auto-resolves types — no frontend change and no extra input needed. Without a resolvable workspace, the table is reported non-fatally intype_resolution_errorsand the build still proceeds.Supporting changes:
get_table_detailnow surfaces the owning bucket'sbackend(needed to pick the INFORMATION_SCHEMA dialect).resolve_workspace_id(backend)callable, so the explicit and auto paths share one code path.Verification
test_semantic_layer_service.py,test_storage_describe_service.py,test_semantic_layer_cli.py;ruff checkclean.build_modelwiring.serve --uiserver: a plainPOST /api/semantic-layer/build(the exact request the UI sends — no workspace, no flags) returned 47 measure / 5 dimension / 1 timestamp withtype_resolution_errors: [], versus all-dimension before.Relationship to #486
Complementary and independent. #486 fixes the classifier (a numeric column maps to the
measurerole); this PR makes the types available for alias/linked tables. Both are needed to fully fix role classification on linked buckets.