fix(objectql)!: floor a plugin driver on an external datasource — the engine applies the query, up to 1000 rows

.changeset/27277-plugin-driver-floor.md

@@ -0,0 +1,30 @@
+---
+'@objectstack/objectql': patch
+'@objectstack/spec': patch
+---
+
+fix(objectql)!: a plugin driver on an external datasource answers every read correctly or refuses: the engine applies the whole query to the rows the driver returns, up to 1000 of them (#27277)
+
+Clause-②: yes (two codes join ERROR_CODE_LEDGER under `@objectstack/objectql`: `EXTERNAL_WRITE_REFUSED` and `EXTERNAL_ROW_CAP_EXCEEDED`, both 422)
+Metadata-protocol: none
+
+<!-- adr-0087: not-required (no-migration-prescription) No key of any schema is added, removed, renamed or re-shaped. A read on a plugin-driver datasource that matches more than 1000 rows is now refused with a 422 that names its own remedy (narrow the filter, or mirror the data with DataSync); there is no metadata for a migration to rewrite. -->
+
+**BREAKING** accept-set narrowing for reads on a plugin-driver external datasource (`@objectstack/objectql`). Only a datasource with both of these is affected: `schemaMode` is not `managed`, and `driver` names a driver a server plugin published (not a built-in id: memory, sqlite, sqlite-wasm, postgres, mysql, mongodb, turso). A managed datasource and an ADR-0015 federated database on a built-in driver are unchanged.
+
+- **Before.** The engine passed the whole query to the plugin driver and served whatever came back. A driver that ignored part of the filter, the sort or the paging answered the wrong rows, and nothing said so.
+- **Now.** The engine holds such a driver behind a correctness floor. The driver is sent `where` and `orderBy`, which it may push down, and `limit: 1001`. It is never sent `offset` or `fields`. The engine then applies the whole query to the rows it gets back (`applyDriverQueryInMemory`): filter, sort (nulls last, and an `id` tie-break on a paged read), offset and limit, and projection. A driver that returned every row still answers the right subset.
+  - `findOne` and `count` are answered from the same read.
+  - `aggregate` is hidden, so the engine aggregates in memory over that read.
+  - The filter is evaluated by the engine's own walker (the one `having` uses). `$like` and `$ilike` now work in its `where` position. It is held to the spec tables `FILTER_LOGIC_CASES`, `FILTER_TEXT_CASES`, `PAGINATION_*_CASES` and `TEMPORAL_CASES`.
+- **Row cap.** A read that matches more than 1000 rows is refused with `EXTERNAL_ROW_CAP_EXCEEDED` (422), whose message says to narrow the filter or mirror the data into a real object with DataSync.
+- **Writes pass through, by id.** The floored driver does not offer `updateMany` or `deleteMany`, so a by-id write under a precondition goes through `update`, and a multi-row predicate update or delete is refused.
+  - A remote refusal for the request's own sake (400, 404, 409, 412 or 422) is `EXTERNAL_WRITE_REFUSED` (422). Its message is the remote's own, and `details.remoteStatus` carries the status.
+  - Any other failure is `ERR_DATASOURCE_UNAVAILABLE` (503, `reason: 'remote'`): no answer, a 5xx, or a 401/403/407/429 credential or rate-limit condition. The cause goes to the server log at `warn` and is never sent to the caller.
+- **One instance.** The engine stores the floored driver in place of the driver, so `getDriverByName`, `datasource()` and the routing for an object all return the same instance. `unwrapExternalDriverFloor` answers the driver behind it.
+- **Re-registration.** A datasource definition registered again without `driver` keeps the driver id stored before it, so the floor stays on.
+- **Not covered.** A `where` with a `ReadableIds` leaf is refused (ADR-0139 D3), because the engine has no semi-join over a plugin driver's rows.
+
+**The remedy.** If a read now meets the cap, narrow its filter, or mirror the data with a `mapping` pulled by a `job` and read the real object.
+
+New exports from `@objectstack/objectql`: `applyDriverQueryInMemory`, `ExternalWriteRefusedError`, `ExternalRowCapExceededError`, their `*_CODE` constants, `EXTERNAL_ROW_CAP`, `EXTERNAL_WRITE_REFUSED_STATUSES`, `needsExternalDriverFloor`, `unwrapExternalDriverFloor`, and the types `DriverQueryInMemory`, `DriverQueryInMemoryOptions`, `ExternalWriteOperation` and `DatasourceUnavailableReason`. `DatasourceDef` gains an optional `driver`.

content/docs/data-modeling/virtual-objects.mdx

@@ -35,8 +35,9 @@ If the remote is a SQL database you can reach, you need no driver of your own 
 built-in one through [External datasources](/docs/data-modeling/external-datasources).
 
 A virtual object pays for its freshness on every request: each list read is a round
-trip to the remote, and the remote's query power is the object's query power. That is
-the trade the table above decides.
+trip to the remote, and a read can match at most 1000 rows
+([the cap](#the-engine-re-applies-every-query)). That is the trade the table above
+decides.
 
 ## The recipe
 
@@ -178,20 +179,27 @@ export function createPortScheduleDriver(transport: PortScheduleTransport): IDat
 }
 ```
 
-**A driver cannot opt out of correctness.** The rules it keeps:
-
-- **Apply every condition, sort and page it receives.** The engine hands the driver
-  the view's filter, the user's search, the sort and the page size, and serves whatever
-  comes back as the answer. A condition the driver drops widens the result silently — a
-  user sees rows the filter excluded, and nothing on the page says so. An engine-side
-  floor for drivers that cannot push a condition down is planned
-  ([#27277](https://github.com/objectstack-ai/objectstack/issues/27277)); until it lands
-  the whole obligation is the driver's.
-- **Refuse by name what the remote cannot run** — an operator, a combinator, a column it
-  does not carry — and fail when the remote fails. Throw; ⛔ never return an empty
-  array, which reads exactly like "no rows match". A thrown driver error answers the
-  caller `500 INTERNAL_ERROR` today; a dedicated refusal for driver failures is planned
-  with the floor above.
+**A driver cannot opt out of correctness, only of performance.** The engine does not
+trust a plugin driver to have applied the query: it re-applies it
+([below](#the-engine-re-applies-every-query)). The rules the driver keeps:
+
+- **Never return fewer rows than match.** The engine filters, sorts and pages the rows
+  your driver returns, so extra rows cost only transfer, but a row your driver leaves out
+  is lost. When you cannot translate part of a `where`, send the remote a WIDER filter,
+  never a narrower one: drop an untranslatable member of a top-level `$and` (or the whole
+  `where`), and drop a whole `$or` or `$not` that contains one. ⛔ Dropping one branch
+  of an `$or` narrows the result. The skeleton above takes the stricter road and refuses
+  instead, which is also safe.
+- **Fail when the remote fails.** Throw; ⛔ never return an empty array, which reads
+  exactly like "no rows match". A read that throws (a refusal by name included) answers
+  the caller `503 ERR_DATASOURCE_UNAVAILABLE` with `reason: 'remote'`. Its message says
+  the remote could not answer and to retry; your error's own text goes to the server log
+  at `warn` and never to the caller. A write the remote refuses for the request's own
+  sake (400, 404, 409, 412 or 422) is answered `422 EXTERNAL_WRITE_REFUSED` carrying your
+  error's message, so set `status` (or `statusCode`) on the error you throw. A write
+  that fails any other way is `503`, as a read is. That includes 401, 403, 407 and 429:
+  an expired credential or a rate limit is the operator's to fix, and its text may name
+  the credential.
 - **Send the remote only the columns it owns.** `update` receives the patch plus any
   field a hook stamped, plus `id`, `updated_at` and `updated_by`, which the engine adds on
   its own and no remote column declares.
@@ -372,8 +380,75 @@ remote row does not carry — so every row reads as someone else's
 `sharingModel: 'public_read_write'` and decide who may edit in `beforeUpdate`, as above.
 Serving an owner column the checks can read is the other way out; the reference test
 does not cover it, so pin it in your own test if you take it.
+
+Missing owner columns fail closed: the edit is refused, never allowed by mistake. Setting
+`sharingModel` is not enough by itself, because an explicit `private` is refused the
+same way. There are two ways out:
+
+- `sharingModel: 'public_read_write'` with a `beforeUpdate` hook that decides, as above;
+- a declared owner field that your driver returns under that field name. A built-in SQL
+  driver maps it through `external.columnMap`; on a plugin driver the mapping is your
+  driver's own job, because the engine does not apply `columnMap`.
+
+On `public_read_write`, a member's delete is still refused: the ownership floor keeps
+deletes by design. A delete passes only when your driver returns `created_by`, or when a
+policy or a share grants it. The validation warning for an external object whose owner
+is unmapped is tracked in
+[#27332](https://github.com/objectstack-ai/objectstack/issues/27332).
 </Callout>
 
+### The engine re-applies every query
+
+For a datasource whose `schemaMode` is not `managed` and whose `driver` is not a built-in
+id (memory, sqlite, sqlite-wasm, postgres, mysql, mongodb, turso), the engine holds the
+driver behind a correctness floor. It does not change a federated database on a built-in
+driver.
+
+- **What the driver is sent.** `find` gets `where` and `orderBy`, which it may push
+  down, and `limit: 1001`. It never gets `offset` or `fields`. `findOne` and `count`
+  reach your driver as that same `find`. `aggregate` is never called; the engine
+  aggregates in memory.
+- **What the engine does with the rows.** It applies the whole query itself: `where`
+  (the engine's own operator vocabulary, `$like` and `$ilike` included), the sort (no
+  value sorts last), `offset` and `limit` (a paged read breaks ties by `id`), and the
+  projection.
+- **Writes** go to your driver unchanged, one record at a time. The floored driver does
+  not offer `updateMany` or `deleteMany`, so no `where` the engine cannot re-check
+  reaches a write. A by-id write under a precondition goes through `update` by id, and
+  a multi-row predicate update or delete is refused.
+
+So a driver grows in three steps, each correct:
+
+1. **Return every row.** Ignore the query and answer the resource's rows. The answer is
+   correct for any resource with at most 1000 rows. Honour `limit` by paging the remote:
+   a remote whose page is smaller than 1001 rows, and that you do not page through,
+   answers fewer rows than match, and the engine cannot detect it.
+2. **Push `where` and `orderBy` down.** The remote returns only the matching rows, so the
+   1000-row cap now counts matching rows, not the whole resource, and less travels. The
+   engine still re-checks every row, so a translation slip shows up as a missing row
+   (step 1's rule), never as an extra one.
+3. **Skip the floor with a conformance-tested driver.** This is planned for a later
+   phase ([#27261](https://github.com/objectstack-ai/objectstack/issues/27261)): a driver
+   that passes the platform's conformance kit would be trusted with paging and the cap.
+   It does not exist yet.
+
+**The 1000-row cap.** A read that matches more than 1000 rows is refused with
+`422 EXTERNAL_ROW_CAP_EXCEEDED` rather than answered from a partial set. The fix is to
+narrow the filter so fewer rows match, or, for data that size, to mirror it into a real
+object with a [scheduled pull](/docs/automation/jobs#pulling-a-mapping) and read that.
+The cap applies to plugin drivers only. It also means a list cannot page past its first
+1000 matching rows: when more than 1000 rows match, every page of that read is refused,
+the one at offset 1000 included. The cap is fixed and not configurable.
+
+**Not covered by the floor.**
+
+- Predicate writes. `updateMany` and `deleteMany` are not offered on a floored datasource,
+  so a write that selects its rows by a filter is refused rather than handed to your
+  driver. Read the ids, then write each one.
+- A read whose scope composes a `ReadableIds` leaf, such as an object
+  `controlled_by_parent`, is refused (ADR-0139): the engine has no semi-join over a
+  plugin driver's rows.
+
 ### Credentials
 
 ⛔ Never put a credential in the stack — not in `config`, not in a hook body, not in an
@@ -389,16 +464,20 @@ repository.
 
 ## What your driver receives
 
-Measured on the reference test, request by request:
+Measured on the reference test, request by request, with the engine's floor
+([above](#the-engine-re-applies-every-query)) in place:
 
 | The platform does | Your driver is called with |
 | :--- | :--- |
 | Boots | `connect()` once. No `syncSchema`, no `introspectSchema` (`checkOnBoot: false`). |
-| Serves a list (`GET /data/berth_slot` with the view's filter, sort and page size) | `find(object, { where, orderBy, limit })`, then `count(object, { where })` for the total |
-| Serves a record (`GET /data/berth_slot/:id`) | `findOne(object, { where: { id }, limit: 1 })` |
-| Writes (`PATCH /data/berth_slot/:id`) | several by-id reads (the pre-image for hooks and checks), then `update(object, id, patch)` with the extra keys above |
+| Serves a list (`GET /data/berth_slot` with the view's filter, sort and page size) | `find(object, { where, orderBy, limit: 1001 })`, then `find(object, { where, limit: 1001 })` for the total. The page size is applied by the engine. |
+| Serves a record (`GET /data/berth_slot/:id`) | `find(object, { where: { id }, limit: 1001 })` |
+| Writes (`PATCH /data/berth_slot/:id`) | several by-id reads as `find` (the pre-image for hooks and checks), then `update(object, id, patch)` with the extra keys above |
 | Refuses a caller with no grant on a list read | nothing |
 
+Your driver should still implement `findOne` and `count`: the engine calls them on a
+driver that is not behind the floor, and the shape test below exercises them.
+
 Two details catch every first driver:
 
 - **`object` is the object name** (`berth_slot`), never `external.remoteName` (`slots`).

content/docs/references/api/automation-api.mdx

@@ -347,7 +347,7 @@ const result = AutomationApiErrorCode.parse(data);
 
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
-| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +347 more>` | ✅ | The inner refusal's code in its WIRE spelling - the ADR-0112 member a record save of the same write answers in `error.code` (e.g. `UNIQUE_VIOLATION`); the closed set `ApiErrorSchema.code` parses against |
+| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +349 more>` | ✅ | The inner refusal's code in its WIRE spelling - the ADR-0112 member a record save of the same write answers in `error.code` (e.g. `UNIQUE_VIOLATION`); the closed set `ApiErrorSchema.code` parses against |
 | **message** | `string` | ✅ | The inner refusal's own diagnostic text, verbatim, without the node and operation prefixes the run's failure text adds |
 
 ### Allowed Values: `FlowInnerRefusal.code`
@@ -496,9 +496,11 @@ const result = AutomationApiErrorCode.parse(data);
 * `EXPORT_NOT_PERMITTED`
 * `EXTERNAL_DATASOURCE_ERROR`
 * `EXTERNAL_IMPORT_ERROR`
+* `EXTERNAL_ROW_CAP_EXCEEDED`
 * `EXTERNAL_SCHEMA_MISMATCH`
 * `EXTERNAL_SCHEMA_MODE_VIOLATION`
 * `EXTERNAL_WRITE_FORBIDDEN`
+* `EXTERNAL_WRITE_REFUSED`
 * `FEEDS_DISABLED`
 * `FIELD_VISIBILITY_UNRESOLVED`
 * `FILES_DISABLED`

content/docs/references/api/contract.mdx

@@ -28,7 +28,7 @@ const result = ApiErrorSchema.parse(data);
 
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
-| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +347 more>` | ✅ | Error code (e.g. VALIDATION_ERROR; StandardErrorCode ∪ the ledger the serving side registers — ERROR_CODE_LEDGER for framework packages) |
+| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +349 more>` | ✅ | Error code (e.g. VALIDATION_ERROR; StandardErrorCode ∪ the ledger the serving side registers — ERROR_CODE_LEDGER for framework packages) |
 | **declaredCode** | `string` | optional | The producer-declared code, verbatim, when it is not a member of the closed `code` vocabulary — the open, author-authored channel (app-specific spellings; ADR-0112) |
 | **message** | `string` | ✅ | Readable error message |
 | **userMessage** | `string` | optional | Producer-marked user-facing refusal text, verbatim. Present exactly when the producer opted in at throw time; consumers render it to end users and keep their generic substitution for anything unmarked. Status-agnostic; never replaces `message`. |
@@ -184,9 +184,11 @@ const result = ApiErrorSchema.parse(data);
 * `EXPORT_NOT_PERMITTED`
 * `EXTERNAL_DATASOURCE_ERROR`
 * `EXTERNAL_IMPORT_ERROR`
+* `EXTERNAL_ROW_CAP_EXCEEDED`
 * `EXTERNAL_SCHEMA_MISMATCH`
 * `EXTERNAL_SCHEMA_MODE_VIOLATION`
 * `EXTERNAL_WRITE_FORBIDDEN`
+* `EXTERNAL_WRITE_REFUSED`
 * `FEEDS_DISABLED`
 * `FIELD_VISIBILITY_UNRESOLVED`
 * `FILES_DISABLED`

content/docs/references/api/error-code-ledger.mdx

@@ -351,9 +351,11 @@ const result = ErrorCode.parse(data);
 * `EXPORT_NOT_PERMITTED`
 * `EXTERNAL_DATASOURCE_ERROR`
 * `EXTERNAL_IMPORT_ERROR`
+* `EXTERNAL_ROW_CAP_EXCEEDED`
 * `EXTERNAL_SCHEMA_MISMATCH`
 * `EXTERNAL_SCHEMA_MODE_VIOLATION`
 * `EXTERNAL_WRITE_FORBIDDEN`
+* `EXTERNAL_WRITE_REFUSED`
 * `FEEDS_DISABLED`
 * `FIELD_VISIBILITY_UNRESOLVED`
 * `FILES_DISABLED`

packages/objectql/src/core.ts

@@ -72,6 +72,7 @@ export type {
   DriverHealth,
   DatasourceUnavailableInfo,
   DatasourceUnavailableKind,
+  DatasourceUnavailableReason,
 } from './driver-connect-errors.js';
 
 // In-memory aggregation fallback