feat: add jdbc-ingestion lifecycle MCP tools (DEV-202506)#93
Draft
DarySchaekenCollibra wants to merge 11 commits into
Draft
feat: add jdbc-ingestion lifecycle MCP tools (DEV-202506)#93DarySchaekenCollibra wants to merge 11 commits into
DarySchaekenCollibra wants to merge 11 commits into
Conversation
…es, database config, ingestion) Adds 14 new tools covering the full flow for DEV-202506 (epic DEV-196736): edge site/capability-type discovery, community/domain scaffolding, connection and capability creation (with driver/file upload and open-ended vendor property support), connection testing, database registration and schema sync configuration, ingestion triggering, and job status polling. Also adds get_data_source_setup_guide, which looks up driver class, connection string format, and per-auth-method connection properties for any of Collibra's ~68 supported data sources by parsing Collibra's public documentation site — avoiding the need to hand-encode each connector's auth requirements.
…sting Adds five new tools surfaced as gaps during live end-to-end testing: find_users, find_domain_types, and find_connections (name-based lookups for ownerIds, create_domain's typeId, and picking up a manually-created connection), plus get_catalog_job_status (polls start_ingestion's job, a separate id space from the existing Edge-job get_job_status). Also: documents configure_database's undocumented capability prerequisite and rewords its discovery error messages accordingly, rejects empty ownerIds early instead of surfacing a raw backend error code, adds a query filter to list_capability_types so it no longer returns every type's full manifest unfiltered, exposes registerSourceTags/ingestSemanticViews/registerDataUsageStatistics on configure_database's schema sync rules, and removes create_connection's now-redundant driver-jar-specific upload convenience. Reworks upload_file's large-file handling: contentBase64 has a hard ceiling since content must be generated by the calling model as a tool argument. Adds a filePath input for reading files directly off local disk, gated at registration time (chip.ServerToolConfig.AllowLocalFileUpload, wired only through cmd/chip's own config) rather than a runtime check, so chip-service's shared registration code has no path that could ever expose it. Documents the full reasoning, including the ruled-out alternatives, in the package doc comment.
Sets Permissions on the tools that call edge-management's connection, capability, and file-upload endpoints so chip-service's token exchange narrows to dgc.edge-integration-capability-manage (writes) or dgc.edge-view-connections-and-capabilities (reads), traced against EdgeAuthorizationHelper and AccessTokenGrantedAuthoritiesConverterTest. Catalog/DGC-core tools are left unscoped pending verification of that separate authorization surface.
Adds a collibra/jdbc-ingestion skill covering the full connect → register → run sequence, the two separate job-id spaces, the Marketplace-only driver policy, and the caution against using a raw run-capability tool for jdbc-ingestion instead of start_ingestion. Cross-links it from the discovery and index skills, and backfills README's tool list, which never listed this tool family.
Adds a schemaNames guard mirroring the existing databaseName ambiguity check, and drops the silent include="*" default, so a multi-schema database can no longer be synchronized wholesale by omission. Updates the jdbc-ingestion skill to instruct agents to confirm scope with the user before calling, rather than defaulting to everything.
A query matching capability types but zero connection types (or vice versa) left the unmatched side as a nil slice, which marshals to JSON null and fails the tool's own array-typed output schema. Initialize both filtered slices as empty instead of nil.
… jobs) Migrate the unique tools from feat/create-tool-for-integration onto the jdbc-ingestion branch, dropping everything that duplicates the existing lifecycle tools. Adds 23 tools: - edge_* capability/connection/job management (list/find/get/delete/run/ cancel/status-history) — the read/delete/run operations with no counterpart on this branch - catalog_generic_* GENERIC integration config, schedules, and jobs (/rest/catalog/1.0/genericIntegration) - jobs_find (/rest/jobs/v1/jobs) and list_integrations Dropped as duplicates: edge_create_connection, edge_find_connections, edge_test_connection, edge_create_capability, edge_get_job_status, jobs_get, edge_update_connection, edge_update_capability. Edge methods are consolidated into the existing per-entity clients; the duplicate capability/connection and job response types are merged into single EdgeCapability/EdgeConnection and Job types, and progressPercentage is int per job-api.yaml.
Tool renames (clarity + consistent namespaces): - create_connection/create_capability -> edge_create_connection/edge_create_capability - find_connections -> edge_find_connections; list_edge_sites/list_capability_types -> edge_list_* - get_catalog_job_status -> get_job_status (DGC job); get_job_status -> edge_get_job_status (Edge job) - catalog_generic_* -> catalog_etl_*; save_config/delete_config -> save_generic_config/delete_generic_config - descriptions: "GENERIC integration instance" -> "ETL integration" Client consolidation: - per-entity Edge clients (edge_capability/edge_connection/edge_job) - canonical DGC Job type in dgc_job_client; catalog DB-sync response -> CatalogJob New skill collibra/etl-integration: - full lifecycle (edge connection/capability -> catalog_etl generic config -> schedule/run) - runtime schema introspection via catalog_etl_get_schema (no embedded schemas) - DGC prerequisites (domain/community/System) with create-or-reuse prompt - ingestion-path disambiguation vs jdbc-ingestion - 15 per-integration references with linked Collibra product docs Indexes updated: README, SKILLS.md, collibra/index navigator. Ignore .DS_Store.
…_database_schemas configure_database registered the Database asset and configured schema sync rules in one call, calling POST /databases before resolving schema ambiguity. A schema-ambiguity error left the Database asset already created, so retrying with schemaNames set failed with databaseConnectionAlreadyUsed instead of resuming. Splits into register_database (discovers the database, registers it) and configure_database_schemas (discovers schemas, sets sync rules, keyed on register_database's returned databaseConnectionId). Neither has a side effect until its own ambiguity is resolved, so an ambiguity error is now a true no-op, and schema sync rules can be reconfigured independently later without re-registering the database. Also trims internal policy detail (specific competitor names) from the jdbc-ingestion skill's user-facing driver-sourcing guidance, keeping the operational instruction without the implementation reasoning.
DarySchaekenCollibra
force-pushed
the
feature/DEV-202506-jdbc-ingestion-mcp-tools
branch
from
July 8, 2026 08:40
3a30eff to
c7f1837
Compare
…ams for lineage capabilities The Databricks and Dataplex lineage capabilities push to the Collibra Data Lineage (Techlin) server and require techlinHost/techlinKey custom parameters to be set before the first run. - databricks-lineage.md: add mandatory customParameters (techlinHost/techlinKey) to Capability parameters + gotcha. - dataplex-lineage.md: add Capability parameters section with the same mandatory custom params + gotcha. - thoughtspot.md: document explicit type / secret requirement for direct (non-DGC-routed) Techlin submission.
…ge-docs docs(etl-skill): document mandatory techlinHost/techlinKey custom params for lineage capabilities
EricWarnerCollibra
requested changes
Jul 10, 2026
EricWarnerCollibra
left a comment
EricWarnerCollibra
left a comment
Contributor
There was a problem hiding this comment.
Overall, in addition to the comments, there is a bit of work required for this to function as intended. This includes:
- Assessment of all tool names and descriptions. As of now, the tool descriptions are vague and would prove challenging for an LLM to reliably use in conjunction with the other tools
- Requirement of some evaluation set to determine how tools are reliably chained together to solve directed business problems. Skills are useful in this context, but the individual tools will be present for non-skill calls as well
Suggest following up directly and we can partner on a joint plan for the above, or happy to hear other opinions
| // AdditionalProperty is one entry of an open-ended, connection-type-defined property | ||
| // list (DGC's ConnectionManifest.ParameterType.USER_DEFINED). Confirmed against a real | ||
| // working Snowflake-via-Generic-JDBC connection: each entry is {name, type, value, | ||
| // secret}, and a FILE-type entry's value is the bare artifact URI (same shape as any |
Contributor
There was a problem hiding this comment.
As discussed, the need to send a secret via a local or remote MCP server will be a sever blocker towards any adoption. Upon review and discussion, my advice is to remove this tool for a first implementation and any others that require a secret within chip. Much better would be to create the connection in platform. @MieszkoWasniowski please add anything I might have missed.
| Error string `json:"error,omitempty" jsonschema:"error message if deletion failed"` | ||
| } | ||
|
|
||
| func NewTool(collibraClient *http.Client) *chip.Tool[Input, Output] { |
Contributor
There was a problem hiding this comment.
IMO the tool description should strongly encourage a user understands the impact of the action being performed prior to deleting an edge connection, as this is a fairly large impact activity and the description of the tool is quite vague
|
|
||
| // NewTool registers upload_file without filePath support (contentBase64 only). Used | ||
| // whenever AllowLocalFileUpload is false — always true for chip-service. | ||
| func NewTool(collibraClient *http.Client) *chip.Tool[Input, Output] { |
Contributor
There was a problem hiding this comment.
Repeating above comment for completeness:
As discussed, the need to send a secret via a local or remote MCP server will be a sever blocker towards any adoption. Upon review and discussion, my advice is to remove this tool for a first implementation and any others that require a secret within chip. Much better would be to create the connection in platform. @MieszkoWasniowski please add anything I might have missed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What does this PR do?
Adds end-to-end CHIP tooling for the Integration MCP epic (DEV-196736), covering
jdbc-ingestion (ticket DEV-202506), Edge management, and generic/ETL integrations
(Dataplex, Databricks Unity Catalog, Purview, Sigma, etc. — via #92 and #94 from
@volodymyr-bodnar, now merged into this branch). An agent can create Edge
connections/capabilities, register a database, configure schema sync, trigger and
poll both jdbc-ingestion and ETL integration runs, and manage existing Edge
capabilities/connections/jobs — without manual UI or API work.
Tool set (40 new tools):
edge_list_sites,edge_list_capability_types,edge_find_connections,find_domain_types,find_users,get_data_source_setup_guide,create_community,create_domain,edge_create_connection,edge_create_capability,upload_file,test_connection,edge_get_job_status,get_job_status,register_database,configure_database_schemas,start_ingestionedge_list_capabilities,edge_find_capabilities,edge_get_capability,edge_delete_capability,edge_run_capability,edge_list_connections,edge_get_connection,edge_delete_connection,edge_cancel_job,edge_get_job_status_historycatalog_etl_get_config,catalog_etl_save_generic_config,catalog_etl_delete_generic_config,catalog_etl_get_schema,catalog_etl_get_schedule,catalog_etl_add_schedule,catalog_etl_update_schedule,catalog_etl_delete_schedule,catalog_etl_get_all_schedules,catalog_etl_cancel_job,catalog_etl_start_jobjobs_find,list_integrationsSkills:
collibra/jdbc-ingestionandcollibra/etl-integration(15per-integration reference docs). Both instruct the agent to confirm which
ingestion path applies before proceeding when a data source supports more than
one, and to always confirm database/schema/table scope with the user rather than
defaulting to "sync everything."
Key design points:
register_database/configure_database_schemasare split so a schema-ambiguityerror can never leave an orphaned, half-registered Database asset behind —
neither tool has a side effect until its own ambiguity is resolved.
upload_filetakes file bytes/local path only.Permissionsscopes traced againstedge-managementsource (notguessed) and applied to the edge-management-backed tools. Catalog/DGC-core-backed
tools remain unscoped pending verification of that separate authorization
surface.
DRAFT: Integrations (Edge management + generic/ETL, #92/#94) are now merged
in. Lineage support is the remaining piece before this is ready for full review.
✅ Checklist