Skip to content

Add provider-normalized context views#12

Open
fivetran-davefowler wants to merge 3 commits into
mainfrom
agent_schema_views
Open

Add provider-normalized context views#12
fivetran-davefowler wants to merge 3 commits into
mainfrom
agent_schema_views

Conversation

@fivetran-davefowler
Copy link
Copy Markdown

@fivetran-davefowler fivetran-davefowler commented May 31, 2026

Summary

Adds information-schema-like context views. v1 deliberately extends only the surfaces INFORMATION_SCHEMA already has — AGENTS.TABLES and AGENTS.COLUMNS — rather than inventing new cross-provider object types.

  • Each provider publishes a normalized AGENTS.<PROVIDER>_TABLES / AGENTS.<PROVIDER>_COLUMNS view (dbt, LookML, OSI).
  • AGENTS.TABLES / AGENTS.COLUMNS take the native INFORMATION_SCHEMA view as the row spine via SELECT t.* and left join every provider view that exists by object identity, appending each provider's columns under a <provider>_ prefix.

Design

  • SELECT t.* spine — inherits whatever native columns the account exposes; no hardcoded INFORMATION_SCHEMA column list (removes the earlier is_hybrid/last_ddl/etc. fragility).
  • Generic identity merge — providers are discovered, not hardcoded; within a provider, rows are aggregated to one per identity to prevent fanout; prefixed columns mean providers never collide.
  • No hardcoded memory counts — memory participates purely additively: when a memory provider later ships its own *_TABLES/*_COLUMNS view, its columns appear automatically with no change here. (Tracked separately; not in this PR.)
  • Fail-soft — view creation runs at the end of each ingestion but a view error warns and is skipped, never failing the data load.
  • Deferred: RELATIONSHIPS / METRICS / ENTITIES. Those are object types semantic providers (OSI) already model; the information-schema-faithful home for relationships is the REFERENTIAL_CONSTRAINTS / KEY_COLUMN_USAGE family (future), not a custom view.

Known limitation

INFORMATION_SCHEMA is per-database, so AGENTS.TABLES/COLUMNS cover the database that holds the AGENTS schema. Multi-database coverage (via ACCOUNT_USAGE) is noted as an open question in the proposal.

Verification

uv run python -m unittest discover -s tests   # 19 tests, OK
uv run python -m compileall -q src
git diff --check                              # clean

Note: tests assert generated-SQL structure; they do not execute against Snowflake.

🤖 Generated with Claude Code

fivetran-davefowler and others added 2 commits May 30, 2026 23:12
- scope to the surfaces information_schema already has (TABLES, COLUMNS); drop
  RELATIONSHIPS/METRICS/ENTITIES and their provider views (deferred; relationships
  should later extend REFERENTIAL_CONSTRAINTS/KEY_COLUMN_USAGE, not a custom view)
- AGENTS.TABLES/COLUMNS use SELECT t.* over information_schema as the spine
  (no hardcoded native column list; inherits whatever the account exposes)
- generic identity merge: left join every discovered {provider}_tables/_columns
  view, enrichment columns appended under a <provider>_ prefix, aggregated to one
  row per identity to prevent fanout; no hardcoded providers
- remove hardcoded memories_count/warnings_count; memory participates later by
  publishing its own *_TABLES/*_COLUMNS view and is picked up automatically
- view creation is fail-soft (warn, never break ingestion)
- align osi_columns identity parsing with osi_tables; rename helper to _relation_identity_sql
- update SPEC, proposal (v1 scope + resolved decisions), root entries, and tests

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- comment that <provider>_ prefixing keeps t.* from colliding with enrichment
- is_time_dimension only true for dimension_group with type time (not duration)
- document name-based, case-folded identity matching in _merge_on
- README: note AGENTS.TABLES/COLUMNS are enriched information_schema and are
  per-database (point workflows at the data's database)
- proposal: reconcile Duplicate And Merge Policy with the v1 prefixed-merge model

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant