fix(cloud-connection): an install-local install connects and records the package's declared datasources

.changeset/28039-install-local-declared-datasources.md

@@ -0,0 +1,29 @@
+---
+'@objectstack/runtime': patch
+'@objectstack/cloud-connection': patch
+---
+
+fix(cloud-connection)!: an install-local install connects and records the datasources the package declares, as a boot of its artifact does
+
+Clause-②: yes (narrowing)
+Metadata-protocol: none
+
+<!-- adr-0087: not-required (no-migration-prescription) No metadata moves: no spec key, authorable spelling, object or datasource definition, stored row or stored shape is added, removed, renamed or re-shaped, and no exported declaration is removed or narrowed, so there is nothing for `objectstack migrate meta` to rewrite. What narrows is the install-local door's accept set: a package whose declared datasource name is already held in the target environment is refused `409` where it was installed `200`. Whether a package is refused depends on what the environment already holds, not on the package's metadata, and the package's metadata stays valid as written; the remedy is the author's choice of a datasource name that is not taken, never a mechanical rewrite of a package source or a stored row. The other categories are closed on facts: both packages publish (not unpublished); no ADR-0087 id is named or touched (not registered or already-registered); and nothing exported is removed or narrowed, only added (not runtime-interface-only, type-surface-only or runtime-export-retired). -->
+
+**BREAKING** accept-set narrowing at `POST /api/v1/marketplace/install-local`, alongside the widening below (a new `@objectstack/runtime` export and a new field on the install answer).
+
+- **Before.** A package declaring a datasource whose name the environment already held (the host's `default`, a datasource created in Setup, another package's) installed `200`, and its declaration was dropped: measured on `main` with a package declaring `showcase_external`, the install answered `200` with `hotLoaded: true`.
+- **Now.** The same install is refused `409 RESOURCE_CONFLICT`, with `details.datasource` naming the held name, before anything is written. Accepting it now would serve the package's objects from the holder's pool (the connect is idempotent on the name) and, by recording the name as code, make the holder's own datasource read-only.
+- **Unchanged.** A package that declares no datasource, or only names nothing else holds, installs as before; a reinstall of the same package is not a collision with itself.
+- **Who sees it.** Only an install-local caller (`os package install`, an AI agent, a script) installing a package whose datasource name collides with one the environment already has.
+
+**The remedy.** Give the package's datasource a name the environment does not already hold, and install again.
+
+**What else changes.**
+
+A package installed through `POST /api/v1/marketplace/install-local` had its objects registered and nothing of its datasources: an object bound to the package's own datasource answered `404 OBJECT_NOT_FOUND` on every read and write, and the datasource was not served as code-defined. Now the install, and every restart's rehydrate of the ledger, take the path a boot of the package's artifact takes:
+
+- **One path.** `@objectstack/runtime` exports `connectPackageDatasources`, which `AppPlugin.start` now calls for the booted artifact and install-local calls for an installed package. Each declared datasource is registered in the metadata registry as `origin: 'code'`, stamped with the package (`_packageId`, `_packageVersion`), its name is recorded in the host's code-datasource set, and it is connected through the `datasource-connection` service's `connectDeclared` (the ADR-0062 D2 gate and the host's connect policy apply as on a boot). The boot's own behaviour is unchanged.
+- **The install answer** carries `datasources`: each declared datasource's connect verdict (`connected`, or the connection service's own non-`ok` status). Omitted for a package that declares none.
+- **A credential that cannot be resolved** is that datasource's named non-`ok` verdict, never a driver gap: the package is installed, Setup → Datasources reports the datasource as `error` with its reason, and a read of an object bound to it answers `503 ERR_DATASOURCE_UNAVAILABLE` naming it. Where a code datasource's credential comes from is not changed here.
+- **Reinstall and uninstall.** A reinstall replaces the previous version's datasources; the uninstall closes their pools, removes their registrations and their names, reported as the `runtime.package-datasources` entry in `cleanups`.

content/docs/data-modeling/external-datasources.mdx

@@ -186,6 +186,17 @@ connect, so queries against its objects keep failing until the server is
 restarted. Do not set it in production.
 </Callout>
 
+<Callout type="info">
+**A package installed into a running runtime** (`os package install`, which calls
+`POST /api/v1/marketplace/install-local`) gets the same treatment at install time
+and at every restart: its declared datasources are registered as `origin: code`,
+read-only, and auto-connected by the same gates. There is no boot left to refuse,
+so a datasource that cannot be connected is installed with its failed `status`
+instead, and reads of its bound objects answer `503 ERR_DATASOURCE_UNAVAILABLE`.
+The install answer lists each datasource's verdict under `datasources`. A package
+declaring a datasource name the runtime already holds is refused `409`.
+</Callout>
+
 ### Seeing the state of a datasource
 
 Every connect attempt's verdict is retained, so a datasource that is down no

packages/cloud-connection/src/marketplace-install-local-plugin.ts

@@ -632,6 +632,19 @@ function manifestIdOf(p: any): string | undefined {
     return p?.manifest?.id ?? p?.id ?? p?.manifest?.name ?? undefined;
 }
 
+/**
+ * [#28039] The datasources a package body declares, as `AppPlugin` reads a
+ * stack's: a list, or a map keyed by name.
+ */
+function declaredDatasources(body: unknown): any[] {
+    const defs = isPlainObject(body) ? body.datasources : undefined;
+    return Array.isArray(defs)
+        ? defs
+        : isPlainObject(defs)
+            ? Object.entries(defs).map(([name, def]) => ({ name, ...(def as any) }))
+            : [];
+}
+
 /**
  * The seed datasets a cached manifest bundles: its `data[]` entries that name
  * an object and carry records. One reading for the two doors that match seed
@@ -1009,6 +1022,10 @@ export class MarketplaceInstallLocalPlugin implements Plugin {
                     const ql: any = ctx.getService('objectql');
                     if (ql && typeof ql.syncSchemas === 'function') await ql.syncSchemas();
                 } catch { /* non-fatal */ }
+                // [#28039] Connect the package's declared datasources — the
+                // boot artifact's were connected in `AppPlugin.start`, and this
+                // package arrives after it.
+                await this.connectDeclaredDatasources(ctx, converted, entry.manifestId, entry.version);
                 // [#21321] Bind the package's script-action bodies and body
                 // hooks — `register` above makes them declared, not runnable.
                 await this.bindArtifactHandlers(ctx, converted, entry.manifestId);
@@ -1788,6 +1805,27 @@ export class MarketplaceInstallLocalPlugin implements Plugin {
             }, 409);
         }
 
+        // 2a. [#28039] The package's datasources are registered as code and
+        //     connected in step 4b′, so a name another holder already owns —
+        //     the host's `default`, a datasource created in Setup or another
+        //     package's — is refused here, before any write: connecting would
+        //     hand the package's objects that holder's pool (the connect is
+        //     idempotent on the name), and recording the name as code would
+        //     make the holder's own datasource read-only to the admin door.
+        const heldDatasource = await this.findHeldDatasourceName(ctx, manifest, manifestId);
+        if (heldDatasource) {
+            return c.json({
+                success: false,
+                error: {
+                    code: 'RESOURCE_CONFLICT',
+                    message: `Package '${manifestId}' cannot be installed: it declares the datasource '${heldDatasource.name}', `
+                        + `and ${heldDatasource.holder} already holds that name in this system. Rename the package's datasource, `
+                        + 'or remove the one holding the name first.',
+                    details: { datasource: heldDatasource.name },
+                },
+            }, 409);
+        }
+
         // 2b. [ADR-0120 D5e] `isolated`-posture gate on installation-wide
         //     uniques. Runs BEFORE hot-register and before anything is written
         //     to the ledger: a stopped install must leave the runtime exactly as
@@ -1967,6 +2005,16 @@ export class MarketplaceInstallLocalPlugin implements Plugin {
             ctx.logger?.warn?.(`[MarketplaceInstallLocal] syncSchemas failed for ${manifestId}: ${err?.message ?? err}`);
         }
 
+        // 4b′. [#28039] Register, record and connect the package's declared
+        //     datasources through the runtime's ONE path — the call
+        //     `AppPlugin.start` makes for a boot artifact — after the schema
+        //     sync, where the boot connects them too. Only for a package the
+        //     running kernel holds: on the lenient path nothing of it is
+        //     registered until the restart's rehydrate, which connects them.
+        const datasources = hotLoadError === undefined
+            ? await this.connectDeclaredDatasources(ctx, converted, manifestId, version)
+            : undefined;
+
         // 4c. [#21321] Bind the package's script-action bodies and body hooks
         //     through the runtime's ONE binder — the call `AppPlugin.start`
         //     makes for a boot artifact. A reinstall replaces the previous
@@ -2026,6 +2074,11 @@ export class MarketplaceInstallLocalPlugin implements Plugin {
                 // [#27237] Omitted, not emptied, for a package whose connectors
                 // name no credential, so that answer's key set is unchanged.
                 ...(credentialRefs !== undefined ? { credentialRefs } : {}),
+                // [#28039] Each declared datasource's connect verdict — `connected`,
+                // or the connection service's named non-`ok` status (a credential
+                // it cannot resolve is `failed-credentials`). Omitted for a package
+                // that declares none, so that answer's key set is unchanged.
+                ...(datasources !== undefined ? { datasources } : {}),
                 ...(unsetRefs.length > 0 ? { credentialNote: installedCredentialNote(unsetRefs, { door }) } : {}),
                 upgradedFrom: conflict === 'marketplace' ? 'previous-marketplace-version' : null,
                 translationsLoaded: seededSummary.translationsLoaded,
@@ -3477,6 +3530,111 @@ export class MarketplaceInstallLocalPlugin implements Plugin {
         return typeof ql?.convertPackageBodyForward === 'function' ? ql.convertPackageBodyForward(manifest) : manifest;
     };
 
+    /**
+     * [#28039] Register, record and connect the datasources `manifest`
+     * declares, through `connectPackageDatasources` — the ONE path, which
+     * `AppPlugin.start` takes for the artifact the host boots: each registered
+     * in the metadata registry as `origin: 'code'` stamped with this package,
+     * its name recorded in the host's code-datasource set, and connected
+     * through the `datasource-connection` service (the D2 gate and the host's
+     * connect policy apply as on a boot). A reinstall replaces the previous
+     * version's datasources; the uninstall withdraws them through the
+     * `runtime.package-datasources` uninstall cleanup.
+     *
+     * Before this, the door registered a package's objects and nothing of its
+     * datasources: an object bound to one had no driver, and the admin door
+     * did not serve the datasource as code-defined.
+     *
+     * ⛔ Where a declared credential comes from is not decided here (#27687):
+     * a credential the connection service cannot resolve is its
+     * `failed-credentials` verdict, which the datasource answers and a read of
+     * a bound object names (`ERR_DATASOURCE_UNAVAILABLE`).
+     *
+     * Answers each declared datasource's verdict, or `undefined` when the
+     * package declares none (or the runtime predates the path). Never throws:
+     * the package is installed either way.
+     */
+    private connectDeclaredDatasources = async (
+        ctx: PluginContext,
+        manifest: unknown,
+        manifestId: string,
+        version: string | undefined,
+    ): Promise<Array<{ name: string; status: string }> | undefined> => {
+        const declared = declaredDatasources(manifest);
+        let connect: typeof import('@objectstack/runtime')['connectPackageDatasources'] | undefined;
+        try {
+            const mod: any = await import('@objectstack/runtime');
+            if (typeof mod?.connectPackageDatasources === 'function') connect = mod.connectPackageDatasources;
+        } catch { /* reported below */ }
+        if (!connect) {
+            if (declared.length > 0) {
+                ctx.logger?.warn?.(
+                    `[MarketplaceInstallLocal] this runtime has no connectPackageDatasources — the datasources of ${manifestId} are NOT connected: `
+                    + 'the objects bound to them have no driver. Upgrade @objectstack/runtime alongside @objectstack/cloud-connection.',
+                );
+            }
+            return undefined;
+        }
+        const objects = isPlainObject(manifest) && Array.isArray(manifest.objects) ? manifest.objects : [];
+        try {
+            // Called with an empty list too: a reinstall whose new version
+            // declares none withdraws what the previous version registered.
+            const verdicts = await connect(ctx, {
+                appId: manifestId,
+                source: 'MarketplaceInstallLocal',
+                datasources: declared,
+                owned: declared.map((datasource) => ({
+                    datasource,
+                    owner: { packageId: manifestId, ...(typeof version === 'string' ? { packageVersion: version } : {}) },
+                })),
+                objects,
+                installed: true,
+            });
+            return declared.length > 0 ? verdicts : undefined;
+        } catch (err: any) {
+            ctx.logger?.error?.(
+                `[MarketplaceInstallLocal] the datasources of ${manifestId} could not be registered or connected: ${err?.message ?? err}`,
+            );
+            return declared.length > 0 ? [] : undefined;
+        }
+    };
+
+    /**
+     * [#28039] The first datasource `manifest` declares whose name another
+     * holder already owns, with that holder: the host's `default`, or a
+     * datasource in the metadata service not stamped with this package (one
+     * created in Setup, the boot artifact's, another package's). A datasource
+     * this package's earlier install registered is its own, so a reinstall
+     * passes. `undefined` when nothing collides or nothing can be read.
+     */
+    private findHeldDatasourceName = async (
+        ctx: PluginContext,
+        manifest: unknown,
+        manifestId: string,
+    ): Promise<{ name: string; holder: string } | undefined> => {
+        const names = declaredDatasources(manifest)
+            .map((ds) => ds?.name)
+            .filter((name): name is string => typeof name === 'string' && name.length > 0);
+        if (names.length === 0) return undefined;
+        if (names.includes('default')) return { name: 'default', holder: "the host's primary datasource" };
+        let metadata: { get?: (type: string, name: string) => Promise<unknown> } | undefined;
+        try { metadata = ctx.getService('metadata'); } catch { return undefined; }
+        for (const name of names) {
+            let item: unknown;
+            try { item = await metadata?.get?.('datasource', name); } catch { continue; }
+            if (!isPlainObject(item)) continue;
+            const owner = item._packageId;
+            if (owner === manifestId) continue;
+            return {
+                name,
+                holder: typeof owner === 'string' && owner.length > 0
+                    ? `the package '${owner}'`
+                    : item.origin === 'runtime' ? 'a datasource created in Setup' : 'a datasource',
+            };
+        }
+        return undefined;
+    };
+
     /**
      * [#21321] Bind an installed package's executable handlers — its
      * `type: 'script'` action bodies and its body hooks — through