diff --git a/.changeset/27786-skills-external-connector.md b/.changeset/27786-skills-external-connector.md new file mode 100644 index 00000000000..b12e32f9d0e --- /dev/null +++ b/.changeset/27786-skills-external-connector.md @@ -0,0 +1,12 @@ +--- +'@objectstack/skills': patch +--- + +docs(skills): objectstack-data teaches `external.connector`, the datasource credential channel that borrows a declared connector instance's credential, beside `credentialsRef` + +Clause-②: no +Metadata-protocol: none + +- **Before.** A datasource can name a declared connector instance in `external.connector` and borrow that instance's credential (ADR-0015 amendment D7), so one credential serves the connector's flow actions and an API-backed datasource's records. `objectstack-data`'s `rules/datasources.md` taught only `credentialsRef`, so an agent following it bound a second copy of a vendor credential the stack already held, and had no way to know the key exists or where it is refused. +- **Now.** The rule's credentials section names the two channels and that exactly one is written: `credentialsRef`, a secret-store handle resolved at connect; or `connector`, for a datasource on a plugin driver such as `@objectstack/driver-odata`, naming a `connectors` instance with a `provider` and an `auth` other than `none`, shown as `schemaMode: 'external', external: { connector: 'erp_api' }`. It says where `connector` is refused (beside `credentialsRef`, on a built-in driver, on a `managed` datasource, and naming an undeclared instance, at `defineStack` or else at connect), that a disabled instance or a stack with no automation service leaves the datasource unconnected with the reason, and that a connector credential not set yet answers `503` per request instead of failing the connect. +- The new text is paid for inside the same file: the `field.columnName` history clause, the auto-connect lead-in and the escape-hatch note are shortened, and phrases that restated their neighbours (the `columnMap` direction the example already shows, the mapping-routing tail, the write-gate tail) are dropped. The file stays under its token ceiling, which does not move. No schema, key or runtime behaviour changes. diff --git a/content/docs/data-modeling/external-datasources.mdx b/content/docs/data-modeling/external-datasources.mdx index 8d28e60472a..64c20325b87 100644 --- a/content/docs/data-modeling/external-datasources.mdx +++ b/content/docs/data-modeling/external-datasources.mdx @@ -263,6 +263,76 @@ for the datasources listed under [When auto-connect fails](#when-auto-connect-fa and degrades with a warning otherwise. (Credential-less drivers such as SQLite simply have no `credentialsRef`.) +### Borrowing a connector instance's credential + +An **API-backed** datasource, one served by a plugin driver such as the +[OData driver](/docs/data-modeling/odata-datasources), has a second channel. When a +[connector instance](/docs/automation/connectors#declaring-a-provider-bound-instance) +already authorizes the vendor, name it in `external.connector` and the datasource +borrows that instance's credential (ADR-0015 amendment D7). The credential is set once, +where the connector's own is set +([What `credentialRef` resolves through](/docs/automation/connectors#what-credentialref-resolves-through)), +and it serves both the connector's flow actions and the datasource's records. + +{/* os:check */} +```typescript +import { defineStack } from '@objectstack/spec'; + +export default defineStack({ + manifest: { id: 'com.example.sales', name: 'Sales', version: '1.0.0', type: 'app' }, + connectors: [ + { + name: 'erp_api', + label: 'ERP API', + type: 'api', + provider: 'rest', + providerConfig: { baseUrl: 'https://erp.example.com' }, + auth: { type: 'bearer', credentialRef: 'ERP_API_TOKEN' }, + }, + ], + datasources: [ + { + name: 'erp', + driver: 'erp', // the OData plugin's name, i.e. the `driver.erp` service + schemaMode: 'external', + config: {}, + external: { + connector: 'erp_api', // no credentialsRef: exactly one channel + validation: { onMismatch: 'warn', checkOnBoot: false }, + }, + }, + ], +}); +``` + +The instance's `auth` decides what is sent: a bearer token, basic auth, or an API key in +its header or query parameter. The rules: + +- **One channel.** `connector` beside a `credentialsRef` is refused. Keep the one that + should supply the credential. +- **API-backed datasources only.** `connector` is refused on a built-in driver + (`postgres`, `mysql`, `mongodb` and the rest), which takes a database credential: bind + that with `credentialsRef`. It is refused on a `managed` datasource too, since an API's + schema is never ObjectStack's to manage: set `schemaMode: 'external'` (or + `'validate-only'`). +- **An instance that can lend.** The name must match a `connectors` entry of the stack + that has a `provider` and an `auth` other than `none`. A catalog descriptor (no + `provider`) or an instance with no `auth` is refused when the stack is validated, and a + datasource created or edited under Setup → Integrations → Datasources is judged the + same way when it is saved. +- **Fail-closed at connect.** An instance that cannot lend when the datasource connects + (not declared, disabled with `enabled: false`, no `auth`), or a stack with no + automation service (`@objectstack/service-automation` is what lends the credential), + leaves the datasource unconnected with the reason, the same connect failure as an + unresolvable `credentialsRef`. +- **A credential not set yet does not fail the connect.** Each request is answered + `503 ERR_DATASOURCE_UNAVAILABLE`, with the reference named in the server log, until + an administrator sets it; the next request after that reads with it, with no restart. +- **Rotation.** A credential set again in the connector credential store reaches the + next request. One the upstream refuses with a `401` is obtained afresh once and the + request retried, which also picks up a rotated environment variable; a second `401` + answers `503`. + ## 5. Writes (double opt-in) Federation is read-only by default. To allow writes, **both** the datasource and the diff --git a/content/docs/data-modeling/odata-datasources.mdx b/content/docs/data-modeling/odata-datasources.mdx index 9d0b6b1c31f..1e34ad3bfa0 100644 --- a/content/docs/data-modeling/odata-datasources.mdx +++ b/content/docs/data-modeling/odata-datasources.mdx @@ -33,7 +33,8 @@ disagree. - **Each entity set you bind has a key.** The key becomes the record's `id`. An entity type with no usable key is refused at the first request. - **You know how it authenticates.** The driver sends no credential, a bearer token, or - HTTP basic. It does not run an OAuth2 flow ([Credentials](#3-credentials)). + HTTP basic, or, borrowed from a connector instance, an API key too. It does not run an + OAuth2 flow ([Credentials](#3-credentials)). ## 1. Mount the plugin @@ -61,7 +62,7 @@ export default defineStack({ | `name` | `odata` | The datasource it serves. It is also the driver name and the `driver.` service suffix. | | `serviceRoot` | (required) | The service root URL, http or https, with no credentials and no query string. | | `version` | `4.0` | The protocol version. `4.0` is the only value. | -| `auth` | `{ scheme: 'none' }` | `none`, `bearer`, or `basic` with a `username`. The secret is the datasource's `external.credentialsRef`. | +| `auth` | `{ scheme: 'none' }` | `none`, `bearer`, or `basic` with a `username`. The secret is the datasource's `external.credentialsRef`. Leave it `none` when the datasource borrows a connector instance's credential ([Credentials](#3-credentials)). | | `secrets` | (none) | Where `credentialsRef` is resolved: an object with `resolve(credentialsRef)`. When absent, the host's `datasource-secret-resolver` service is used; `os serve` registers one. | | `pageSize` | `500` | Rows asked for per server page (`Prefer: odata.maxpagesize`). | | `requestTimeoutMs` | `30000` | The timeout of each request. | @@ -164,8 +165,17 @@ What each piece does on this driver: ## 3. Credentials ⛔ Never put a credential in the stack, in the plugin options, or in the `serviceRoot` -URL. The secret is the datasource's `external.credentialsRef`, and the plugin resolves it -through the host's secret store: +URL. The datasource names exactly one of two channels: + +- **A secret of its own**, `external.credentialsRef`, resolved through the host's secret + store. This is the example in step 2. +- **A connector instance's credential**, `external.connector`, when a connector already + authorizes the same service + ([Borrowing a connector instance's credential](#borrowing-a-connector-instances-credential)). + +### A secret of its own: `credentialsRef` + +The plugin resolves `external.credentialsRef` through the host's secret store: - **`bearer`** sends `Authorization: Bearer `. - **`basic`** takes `username` from the options and the password from the secret. @@ -197,6 +207,56 @@ the credential: an auth scheme with no `credentialsRef`, a `credentialsRef` with `secrets` nor the service, and a handle that does not resolve. The caller is answered `503 ERR_DATASOURCE_UNAVAILABLE`, and the cause goes to the server log. +### Borrowing a connector instance's credential + +When a [connector instance](/docs/automation/connectors#declaring-a-provider-bound-instance) +already authorizes the ERP for your flows, the datasource can borrow its credential instead +of holding a secret of its own (ADR-0015 amendment D7). One credential, set where the +connector's is set, then serves the connector's actions and the datasource's records. +Name the instance in `external.connector`, leave `credentialsRef` out, and create the +plugin with `auth` left at `none`: + +```typescript +connectors: [ + { + name: 'erp_api', label: 'ERP API', type: 'api', + provider: 'rest', providerConfig: { baseUrl: 'https://erp.example.com' }, + auth: { type: 'bearer', credentialRef: 'ERP_API_TOKEN' }, + }, +], +datasources: [ + { + name: 'erp', driver: 'erp', schemaMode: 'external', config: {}, + external: { connector: 'erp_api', allowWrites: true }, // no credentialsRef + }, +], +plugins: [ + createODataDriverPlugin({ + name: 'erp', + serviceRoot: 'https://erp.example.com/odata/v4/sales/', // no auth, no secrets + }), +], +``` + +- **The instance's `auth` decides how the credential is sent:** `Authorization: Bearer`, + `Authorization: Basic`, or an API key in its `headerName` header (default `X-API-Key`) + or, with `paramName`, a query parameter. +- **The plugin borrows at the first request**, from the `connector-credential-source` + service that `@objectstack/service-automation` registers, and keeps the handle. Every + request then presents the current credential: one past its expiry is obtained afresh + before it is sent, and one the service refuses with a `401` is obtained afresh once and + the request retried. A second `401` answers `503`, naming the instance in the server + log. +- **It fails closed.** Each of these refuses the request with nothing sent, answered + `503 ERR_DATASOURCE_UNAVAILABLE` with the cause in the server log: no automation + service in the stack, an instance that cannot lend (not declared, disabled, no + `auth`), a credential that is not set yet, a `credentialsRef` beside the connector, + and an `auth` scheme other than `none` on the plugin, which would send a second + credential. An instance that cannot lend also fails the datasource's connect, so + Setup → Integrations → Datasources shows why + ([Borrowing a connector instance's credential](/docs/data-modeling/external-datasources#borrowing-a-connector-instances-credential) + has the rules the stack is validated against). + ## 4. What the service is asked Behind the floor, a list read is one `GET` on the entity set: diff --git a/content/docs/data-modeling/virtual-objects.mdx b/content/docs/data-modeling/virtual-objects.mdx index 6a1ad5e9c43..93057bdcebf 100644 --- a/content/docs/data-modeling/virtual-objects.mdx +++ b/content/docs/data-modeling/virtual-objects.mdx @@ -460,7 +460,8 @@ So a driver grows in three steps, each correct: 3. **Skip the floor with a conformance-tested driver.** This is planned for a later phase ([#27261](https://github.com/objectstack-ai/objectstack/issues/27261)): a driver that passes the platform's conformance kit would be trusted with paging and the cap. - It does not exist yet. + The kit ships ([Run the conformance kit](#run-the-conformance-kit)); the trust does + not exist yet. **The 1000-row cap.** A read that matches more than 1000 rows is refused with `422 EXTERNAL_ROW_CAP_EXCEEDED` rather than answered from a partial set. The fix is to @@ -492,6 +493,61 @@ configured by the host that composes it, through the options it passes to the pl keep the credential in the host's secret handling and out of metadata and the repository. +When a [connector instance](/docs/automation/connectors#declaring-a-provider-bound-instance) +already authorizes the vendor your driver calls, the datasource can name it in +`external.connector` instead +([Borrowing a connector instance's credential](/docs/data-modeling/external-datasources#borrowing-a-connector-instances-credential)): +one credential then serves the connector's flow actions and your driver's requests. The +connect step checks that the named instance can lend, and fails the datasource when it +cannot, but it hands your already-built driver nothing. Your driver borrows the +credential itself, from the `connector-credential-source` service that the automation +service registers: + +```typescript +import type { PluginContext } from '@objectstack/core'; +import { + CONNECTOR_CREDENTIAL_SOURCE_SERVICE, + callWithBorrowedCredential, + withoutBorrowedCredential, + type BorrowedConnectorAuth, + type ConnectorCredentialHandle, + type IConnectorCredentialSource, +} from '@objectstack/types'; + +// At the first request: `connector` is the datasource's `external.connector`. +async function borrow(ctx: PluginContext, connector: string): Promise { + const source = ctx.getService(CONNECTOR_CREDENTIAL_SOURCE_SERVICE); + return source.borrow(connector); +} + +// Every request: the helper refreshes an expired credential, and on a 401 obtains it +// afresh once and retries; a second 401 throws a 503 naming the instance. +async function request(handle: ConnectorCredentialHandle, url: string): Promise { + let sent: BorrowedConnectorAuth | undefined; + const response = await callWithBorrowedCredential(handle, (auth) => { + sent = auth; + return fetch(url, { + headers: auth.type === 'bearer' ? { Authorization: `Bearer ${auth.token}` } : {}, + }); + }); + if (!response.ok) { + // An upstream may echo the credential in its error text: remove it before throwing. + const text = await response.text(); + throw Object.assign(new Error(sent ? withoutBorrowedCredential(text, sent) : text), { + status: response.status, + }); + } + return response; +} +``` + +The sketch sends only a bearer token. Present `auth` as its `type` says: `bearer` as +`Authorization: Bearer`, `basic` as `Authorization: Basic`, and `api-key` in its +`headerName` header or, with `paramName`, as a query parameter. Refuse the request, +sending nothing, when the service is absent or the datasource also names a +`credentialsRef`. The OData driver is the reference implementation +(`packages/drivers/driver-odata/src/plugin.ts` and `transport.ts`). + ## What your driver receives Measured on the reference test, request by request, with the engine's floor @@ -604,6 +660,41 @@ operator from "refused" to a shape case, then teach `OPERATORS` the mapping. To same shapes arrive through a booted stack — HTTP door, permissions, formula, hook — start from the reference test named at the top of this page. +### Run the conformance kit + +The shape test pins what *your* driver sends. The platform's plugin-driver conformance +kit pins what the *engine* relies on, from outside this repository: a driver in its own +repository runs it in its own CI, so a platform change to that contract fails there +before it reaches your users. It ships in `@objectstack/objectql`: +`definePluginDriverConformanceSuite` registers one test per case under your runner's +`describe` / `it` (vitest and jest alike), and `runPluginDriverConformance` returns a +report without a test framework. + +```typescript +import { afterAll, describe, it } from 'vitest'; +import { definePluginDriverConformanceSuite } from '@objectstack/objectql'; +import { createPortScheduleDriver } from './port-schedule-driver'; +import { fakeRemoteHolding } from './fake-remote'; // your fake of the remote's API + +definePluginDriverConformanceSuite( + { describe, it, afterAll }, + { + // A driver over a remote holding exactly the fixture's rows, served under its object. + setup: (fixture) => ({ driver: createPortScheduleDriver(fakeRemoteHolding(fixture)) }), + }, +); +``` + +The kit's oracle is the spec's driver conformance tables, the standard every built-in +driver is held to. Each read is checked against the floor: the driver may push all of a +filter down, part of it or none, and may refuse what it cannot translate, but must never +return fewer rows than match, invent a row, or answer a failure as rows. Add +`setupUnavailable` (a remote that is down) for the outage cases and `writeProbes` (one +write the remote accepts, one it refuses) for the write cases; without them those cases +report `not-offered`. `tiers: ['floor', 'direct']` adds the stricter tier, where the +driver answers the whole query itself; the floor does not need it. The OData driver runs the +kit in `packages/drivers/driver-odata/src/odata-driver.conformance-kit.test.ts`. + ## Five business examples **Port berth scheduling — computed, real-time.** The worked example above. The diff --git a/skills/objectstack-data/rules/datasources.md b/skills/objectstack-data/rules/datasources.md index f39634a1381..4f6005a77f3 100644 --- a/skills/objectstack-data/rules/datasources.md +++ b/skills/objectstack-data/rules/datasources.md @@ -1,11 +1,11 @@ # Datasources & Federation -A **datasource** (`defineDatasource`, a `*.datasource.ts` metadata file) is a +A **datasource** (`defineDatasource`, a `*.datasource.ts` file) is a connection to a data store. Objects route to one via their `datasource` field (default: `'default'`). `default` is host-reserved: the host connects it from `OS_DATABASE_URL`, and a stack that declares a datasource named `default` is refused. Declare a datasource under its own name only to read/write a -**separate** or **external** database. +**separate** or **external** store. Full field reference: `node_modules/@objectstack/spec/src/data/datasource.zod.ts`. @@ -37,44 +37,48 @@ ObjectSchema.create({ ### ✅ / ❌ Column mapping (ADR-0062 D7) -- ✅ Map remote columns with **`external.columnMap`** (`remoteColumn → localField`). -- ❌ **`field.columnName` does not exist — on ANY object.** It was removed in the - 16.x line (the SQL driver hardcodes the physical column to the field key, so a - custom name was ignored), and authoring it is a parse error everywhere, not - only on a federated object. +- ✅ Map remote columns with **`external.columnMap`**. +- ❌ **`field.columnName` does not exist — on ANY object.** Authoring it is a + parse error everywhere, not only on a federated object. ## Auto-connect (no `onEnable`) -A declared datasource is built into a live driver, connected, and its federated -objects' read metadata registered **automatically at boot** — no `onEnable` / -`ctx.drivers.register`. It auto-connects when **meaningfully addressed**: +A declared datasource is connected, and its federated objects registered, +**automatically at boot** when it is **meaningfully addressed**: 1. it is **external** (`schemaMode !== 'managed'`), **or** 2. an object **explicitly** binds via `object.datasource === `, **or** 3. it sets **`autoConnect: true`**, **or** 4. a **`datasourceMapping` rule routes at least one object to it**. -A `managed` datasource that nothing routes to stays **metadata-only** — visible but -not connected. +A `managed` datasource nothing routes to stays **metadata-only** (not connected). ⚠️ A `datasourceMapping` rule is **routing, not a hint**. A rule pointing at a datasource that cannot be connected fails the boot, and a query against a mapped -object throws instead of silently resolving the default store. Do not declare a -mapping you do not mean. +object throws. -> `onEnable` + `ctx.drivers.register(driver)` remains supported only as an escape -> hatch for drivers built dynamically at runtime; it is idempotent with auto-connect. +> `ctx.drivers.register(driver)` in `onEnable` is only an escape hatch for a +> runtime-built driver; idempotent with auto-connect. -## Credentials — fail-closed +## Credentials — one channel, fail-closed -Never inline a password. Use `external.credentialsRef` and store the secret in the -secret store; it is resolved **at connect, before the driver is built**. A declared -`credentialsRef` that cannot be resolved/decrypted (or no secret store configured) -leaves the datasource **unconnected with a clear error** — never connected without -the credential. +Never inline a password. `external` names **one** credential channel: + +- **`credentialsRef`** — a secret-store handle, resolved at connect, before the + driver is built. +- **`connector`** — an API-backed datasource (a plugin driver such as + `@objectstack/driver-odata`) borrows the credential of a `connectors` instance + with a `provider` and an `auth` other than `none`: + `schemaMode: 'external', external: { connector: 'erp_api' }`. Refused beside + `credentialsRef`, on a built-in driver, on a `managed` datasource, and naming + an undeclared instance (at `defineStack`, else at connect). + +No secret store, an unresolvable handle, a disabled instance or no automation +service leaves the datasource **unconnected with a clear error**. A connector +credential not set yet answers `503` per request. ## Writes — double opt-in Federation is read-only by default. To write, **both** gates must be on: `datasource.external.allowWrites: true` **and** the object's `external.writable: true`. -With either off, insert/update/delete on the federated object is rejected. +Otherwise every write to the object is rejected.