Skip to content

Unity catalog explorer#1871

Open
misha-db wants to merge 67 commits into
mainfrom
unity-catalog
Open

Unity catalog explorer#1871
misha-db wants to merge 67 commits into
mainfrom
unity-catalog

Conversation

@misha-db
Copy link
Copy Markdown
Contributor

Changes

Unity Catalog Features

Tree View

  • Hierarchical browser: Catalogs → Schemas → Tables / Volumes / Functions → Columns
  • Filterable by typing (filterOnType)
  • Only visible when authenticated and extension is active

Node Types

  • Catalog — name, tooltip with comment
  • Schema — name, tooltip with comment
  • Table — type badge (TABLE / VIEW / MATERIALIZED_VIEW), format (delta, parquet, …), tooltip with owner, timestamps, comment; expandable to columns
  • Volume — type badge (MANAGED / EXTERNAL), tooltip with owner, comment
  • Function — full name in tooltip
  • Column — name, data type, nullable indicator (key icon for non-nullable), comment

Context Menu Actions

  • Copy — copies full qualified name (e.g. catalog.schema.table) for all nodes; copies just column name for columns
  • Copy storage location — copies storage path for tables/volumes that have one
  • Copy view SQL — copies view definition SQL for views and materialized views
  • Open in Databricks — opens the item in the Databricks Explore UI in a browser (not available for columns)
  • Refresh — re-fetches a specific node and its children (not available for columns)

Toolbar

  • Refresh button — refreshes the entire Unity Catalog tree

Error Handling

  • Error nodes displayed inline when a catalog/schema/table listing fails

Tests

UnityCatalogTreeDataProvider.test.ts

@misha-db misha-db requested a review from ilia-db March 31, 2026 15:44
Comment thread packages/databricks-vscode/src/ui/unity-catalog/UnityCatalogTreeDataProvider.ts Outdated
Comment thread packages/databricks-vscode/src/ui/unity-catalog/UnityCatalogTreeDataProvider.ts Outdated
Comment thread packages/databricks-vscode/src/ui/unity-catalog/UnityCatalogTreeDataProvider.ts Outdated
Comment thread packages/databricks-vscode/src/ui/unity-catalog/UnityCatalogTreeDataProvider.ts Outdated
Comment thread packages/databricks-vscode/src/extension.ts
Comment thread packages/databricks-vscode/src/extension.ts
@misha-db misha-db temporarily deployed to test-trigger-is May 12, 2026 18:28 — with GitHub Actions Inactive
@misha-db misha-db temporarily deployed to test-trigger-is May 13, 2026 08:37 — with GitHub Actions Inactive
@misha-db misha-db temporarily deployed to test-trigger-is May 21, 2026 12:20 — with GitHub Actions Inactive
@misha-db misha-db temporarily deployed to test-trigger-is May 25, 2026 11:09 — with GitHub Actions Inactive
@misha-db misha-db temporarily deployed to test-trigger-is May 25, 2026 12:48 — with GitHub Actions Inactive
@misha-db misha-db temporarily deployed to test-trigger-is May 25, 2026 14:11 — with GitHub Actions Inactive
@misha-db misha-db temporarily deployed to test-trigger-is May 25, 2026 17:10 — with GitHub Actions Inactive
) {
this.disposables.push(
this.connectionManager.onDidChangeState(() => {
this.catalogsCache = undefined;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to clean the childrenCache here as well?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, added

return cached;
}
const result = await loader();
this.childrenCache.set(key, result);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A transient API blip turns this into a sticky error state that survives until the user manually refreshes that exact node. Detect error/empty results and skip caching, or attach a TTL here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, skipping caching results with error nodes

schemaName: string
): Promise<UnityCatalogTreeNode[]> {
const [tablesResult, volumesResult, functionsResult, modelsResult] =
await Promise.allSettled([
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For workspaces with thousands of tables in a schema this is a heavy initial load, and loadSchemaChildren (loaders.ts:107) fires four such drains in parallel before the user sees anything. Consider adding a max-items cap with a "load more" sentinel

"databricks.unityCatalog.showDetail",
showDetail
),
treeView.onDidChangeSelection(async (event) => {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance concern for big catalogs:

Selection auto-opens the detail panel on every keyboard arrow — onDidChangeSelection fires on every focus change, not just clicks. Arrow-key browsing through a large tree will repeatedly create + post to the webview, and on schema/table selection will kick off loadNodeEnrichments (4 parallel API calls each). Consider gating this on an explicit user action or introduce a debounce.

}

// eslint-disable-next-line @typescript-eslint/no-explicit-any
const stored: StoredFavoriteNode = {...node} as StoredFavoriteNode;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will carry stale metadata as it deep-copies the whole node and stores it in workspace state.

Owner/comment/format/storageLocation will silently age until the table is re-fetched; renamed or deleted favorites will display indefinitely. Store only {kind, fullName, version?} and rehydrate from a fresh fetch when the Favorites root is expanded.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't found API for loading UC resources in bulk by fullnames, meaning if we rehydrate from a fresh fetch it will be N calls (where N is the number of pinned/favorite nodes). this might not be an issue, assuming most users might have 2-5 favorite resources

}>;
}

async function loadChildrenForNode(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this re-implement loaders.ts:loadSchemaChildren for the no-cache path? Consider deduplicating; otherwise the two implementations will drift

@github-actions
Copy link
Copy Markdown
Contributor

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/vscode

Inputs:

  • PR number: 1871
  • Commit SHA: 64eeddcb8a8e5d55d98ece1bb7d2e22d4cd631f4

Checks will be approved automatically on success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants