Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion SKILLS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,12 @@ those tools. Skill content lives in [`pkg/skills/files/collibra/`](pkg/skills/fi
|---|---|
| `collibra/index` | Navigator β€” start here when unsure which skill applies |
| `collibra/discovery` | Semantic vs keyword search; resolving names to UUIDs |
| `collibra/lineage` | Technical lineage; DGC UUID ↔ lineage entity ID bridge; column-level workaround |
| `collibra/lineage` | Querying existing technical lineage; DGC UUID ↔ lineage entity ID bridge; column-level workaround |
| `collibra/asset-create` | `create_asset` workflow; RICH_TEXT Markdown handling; duplicate gating |
| `collibra/asset-edit` | `edit_asset` operation types |
| `collibra/jdbc-ingestion` | End-to-end jdbc-ingestion setup: Edge connection/capability, database registration, running and polling a sync |
| `collibra/etl-integration` | End-to-end ETL sync integrations (Dataplex, Databricks UC, Purview, Sigma, …): connection, capability, generic config (fetched live), schedule/run; per-integration references |
| `collibra/techlin` | Technical lineage (techlin) harvesting setup: prerequisites (Database asset + jdbc-ingestion), Technical Lineage capability, `start_technical_lineage` trigger; per-source references (Snowflake) |

Each skill is one `SKILL.md` per directory, with frontmatter (`description`, `related`) and an
optional `references/` directory for bundled reference documents.
Expand Down
21 changes: 21 additions & 0 deletions pkg/clients/catalog_database_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,27 @@ func RegisterDatabase(ctx context.Context, client *http.Client, request AddDatab
return &database, nil
}

// GetDatabase fetches a registered Database asset via GET /databases/{databaseId}.
func GetDatabase(ctx context.Context, client *http.Client, databaseID string) (*Database, error) {
endpoint := catalogDatabaseBasePath + "/databases/" + databaseID
var database Database
if err := doCatalogDatabaseGet(ctx, client, endpoint, &database); err != nil {
return nil, err
}
return &database, nil
}

// GetDatabaseConnection fetches a database connection via
// GET /databaseConnections/{databaseConnectionId}.
func GetDatabaseConnection(ctx context.Context, client *http.Client, databaseConnectionID string) (*DatabaseConnection, error) {
endpoint := catalogDatabaseBasePath + "/databaseConnections/" + databaseConnectionID
var connection DatabaseConnection
if err := doCatalogDatabaseGet(ctx, client, endpoint, &connection); err != nil {
return nil, err
}
return &connection, nil
}

// RefreshSchemaConnections asynchronously refreshes the schema connections known for a
// database connection via POST /schemaConnections/refresh?databaseConnectionId=...
func RefreshSchemaConnections(ctx context.Context, client *http.Client, databaseConnectionID string) error {
Expand Down
23 changes: 23 additions & 0 deletions pkg/clients/technical_lineage_client.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
package clients

import (
"context"
"fmt"
"net/http"
)

const technicalLineageBasePath = "/rest/catalog/1.0/technicalLineage"

// StartTechnicalLineageHarvest triggers the technical lineage harvest for a registered
// Database asset via POST /harvester/{assetId}. The endpoint accepts the request with
// 202 and an empty body β€” the DGC job it spawns is not returned and must be located
// through the jobs API afterwards.
func StartTechnicalLineageHarvest(ctx context.Context, client *http.Client, assetID string) error {
endpoint := technicalLineageBasePath + "/harvester/" + assetID
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, nil)
if err != nil {
return fmt.Errorf("failed to create request: %w", err)
}
_, err = executeRequest(client, req)
return err
}
8 changes: 7 additions & 1 deletion pkg/skills/files/collibra/etl-integration/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
description: Set up, configure, schedule, and run Edge ETL integrations (metadata sync capabilities like Dataplex, Databricks Unity Catalog, Purview, Sigma) end to end.
related: collibra/jdbc-ingestion, collibra/discovery
related: collibra/jdbc-ingestion, collibra/discovery, collibra/techlin
---

# ETL integrations β€” connect, configure, schedule, run
Expand Down Expand Up @@ -28,6 +28,12 @@ which ingestion type it should be** β€” don't assume ETL just because this skill
Only proceed with this ETL skill once the user has chosen it (or it's the only supported
path).

**Technical Lineage capabilities are not ETL integrations.** If the capability to create
or run is a technical-lineage one (display name "Technical Lineage for <Source>", e.g.
Snowflake), stop here β€” that whole flow, including creating the capability and its
mandatory parameter conversation, is owned by `collibra/techlin`. Load that skill
instead; do not create the capability from this one.

## The three configuration layers (merged at run time)

When a run starts, DGC's catalog module merges three parameter sets into one and sends
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ conceptual, not field-level.
through the `catalog_etl_*` tools β€” see "Not controllable via chip (yet)" in `SKILL.md`.

Notes:
- **JDBC technical lineage capabilities (e.g. "Technical Lineage for Snowflake") are not
`catalog_etl_*` integrations** β€” they are triggered per Database asset via
`start_technical_lineage`. See `collibra/techlin`.
- ThoughtSpot's manifest uses a generated typeId; confirm the live value with
`edge_list_capability_types` if `edge_create_capability` rejects the id above.
- SAP capabilities (AI Core, Datasphere, Data Products, S/4HANA) are out of scope for this
Expand Down
13 changes: 11 additions & 2 deletions pkg/skills/files/collibra/index/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
description: Navigator for chip's Collibra skills. Start here when unsure which skill applies.
related: collibra/discovery, collibra/lineage, collibra/asset-create, collibra/asset-edit, collibra/data-product-create, collibra/context, collibra/jdbc-ingestion, collibra/etl-integration
related: collibra/discovery, collibra/lineage, collibra/asset-create, collibra/asset-edit, collibra/data-product-create, collibra/context, collibra/jdbc-ingestion, collibra/etl-integration, collibra/techlin
---

# Collibra skills β€” navigator
Expand All @@ -14,13 +14,22 @@ must be bridged to another, and which permissions are required.
| Task | Skill |
|---|---|
| Find data about a topic by meaning (semantic) vs by exact name (keyword) | `collibra/discovery` |
| Trace upstream sources, downstream consumers, or impact of a change | `collibra/lineage` |
| Trace upstream sources, downstream consumers, or impact of a change in **existing** lineage (query only) | `collibra/lineage` |
| Create any new asset (Business Term, Table, Column, KPI, …) | `collibra/asset-create` |
| Modify an existing asset's attributes, relations, tags, status, or owners | `collibra/asset-edit` |
| Register a table (and its dimension tables) as a Collibra Data Product with ports | `collibra/data-product-create` |
| Generate governed YAML context (semantic blueprints, metric definitions) for an asset | `collibra/context` |
| Set up an Edge JDBC connection/capability and run jdbc-ingestion | `collibra/jdbc-ingestion` |
| Set up any metadata-sync ETL integration (Dataplex, Databricks UC, Purview, Sigma, …) β€” connection, capability, generic config, schedule/run | `collibra/etl-integration` |
| **Create, set up, enable, or run technical lineage** for a registered database (Snowflake, …) | `collibra/techlin` |

**"Technical lineage" is ambiguous β€” route by verb, and don't ask the user which skill
to use.** *Querying* what already exists (trace, upstream, downstream, impact, "where
does this come from") β†’ `collibra/lineage`. *Producing* it (create, set up, configure,
enable, run, harvest, ingest lineage for a source) β†’ `collibra/techlin`. If the
request genuinely doesn't say (e.g. just "technical lineage for Snowflake"), ask whether
they want to explore existing lineage or set up harvesting β€” never present the skill
names themselves as the choice.

If a task is a single tool call with no chaining (e.g. `get_asset_details` by UUID,
`list_asset_types`, `pull_data_contract_manifest`), no skill is needed β€” the tool's own
Expand Down
5 changes: 4 additions & 1 deletion pkg/skills/files/collibra/jdbc-ingestion/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
description: Set up an Edge JDBC connection and capability, register a database, and run jdbc-ingestion end to end.
related: collibra/discovery
related: collibra/discovery, collibra/techlin
---

# JDBC ingestion β€” connect, register, and run
Expand Down Expand Up @@ -75,6 +75,9 @@ which ingestion type it should be** β€” don't assume JDBC just because this skil
202/success from `start_ingestion` only means the job was accepted, not that
ingestion finished β€” always poll to confirm completion.

Once ingestion has completed, **technical lineage** can be layered on top of the
registered database β€” see `collibra/techlin`.

## Two job-id spaces β€” do not cross them

- **Edge-site jobs** (from `test_connection`) β†’ poll with **`edge_get_job_status`**.
Expand Down
11 changes: 8 additions & 3 deletions pkg/skills/files/collibra/lineage/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
---
description: Trace upstream sources and downstream consumers in Collibra's technical lineage graph. Covers the DGC UUID β†’ lineage entity ID bridge.
related: collibra/discovery, collibra/index
description: Query the existing technical lineage graph β€” trace upstream sources and downstream consumers. Covers the DGC UUID β†’ lineage entity ID bridge. Read-only β€” for setting up, creating, or running technical lineage harvesting, load collibra/techlin instead.
related: collibra/discovery, collibra/index, collibra/techlin
---

# Technical lineage
# Technical lineage β€” querying the graph

This skill only covers **querying** existing lineage. If the user wants to *set up*,
*create*, *configure*, *enable*, or *run* technical lineage (harvest lineage from a data
source like Snowflake), **this is the wrong skill β€” load `collibra/techlin`**
before doing anything else.

Technical lineage maps the physical data flow β€” including unregistered assets, temporary
tables, and source code β€” across systems. This is broader than business lineage (which only
Expand Down
Loading
Loading