You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 86edcaf
Browse filesBrowse the repository at this point in the historyBrowse files
fix(objectql)!: floor a plugin driver on an external datasource — the engine applies the query, up to 1000 rows (#27403)
<!-- claude-projects-attribution: {"github_login":"marchtian"} -->
_Requested by **田騏** · [project
thread](https://claude.ai/code/project/chan_01VmuUhXU9SSUzqJYyXSmYSi?thread=cmsg_01VmuUhXU9SSUzqJYyXSmYSi1kab1zhM1QA3ViSW5ujLcS)_Fixes#27277
Clause-②: yes (two codes join ERROR_CODE_LEDGER under
`@objectstack/objectql`: `EXTERNAL_WRITE_REFUSED`,
`EXTERNAL_ROW_CAP_EXCEEDED`, both 422)
Metadata-protocol: none
## 维护者速读
- **改了什么:** 插件 driver 背后的外部数据源(`schemaMode` 非 managed,且 `driver` 不是内置
id)现在由引擎兜底。driver 只收到 `where`、`orderBy` 和 `limit:
1001`,引擎自己再把整条查询(过滤、排序、分页、投影)作用在返回的行上。超过 1000 行 ⇒ 422
`EXTERNAL_ROW_CAP_EXCEEDED`;远端对写入回 4xx ⇒ 422
`EXTERNAL_WRITE_REFUSED`;其余远端失败 ⇒ 503 `ERR_DATASOURCE_UNAVAILABLE`。
- **为什么改:** `DriverCapabilities` 没有任何下推声明(17.0.0 已退役),引擎无从得知插件 driver
执行了查询的哪一部分。一个忽略过滤的 driver 会静默返回错误的行。按裁决 Q1=B / Q2=A′ / Q3=A′ 落地。
- **风险与回滚:** 内置 driver、managed 数据源、ADR-0015 联邦 SQL 数据源都不受影响(测试钉住
`getDriverByName` 仍返回原实例)。受影响的只有插件 driver 外部对象:匹配超过 1000 行的读取从「静默返回」变成
422,变更集按 BREAKING 申报。回滚即 revert 两个提交。
- **先看:** `packages/objectql/src/external-driver-floor.ts`(Proxy
包装、上限读取、写入错误映射),以及 `engine.ts` 的
`applyExternalDriverFloor`(两个注册入口都调用它,drivers 映射里只放一个实例)。
- **两分钟验证:** `pnpm --filter @objectstack/objectql exec vitest run
--project local src/external-driver-floor.test.ts
src/driver-query-in-memory.conformance.test.ts`,预期 123 passed。
- **需要你做的:** spec 提交(账本两行)等 epic PM 的完整 Fable 审查。Acceptance notes 第 1、2
条是开放项,需要定夺是否另开卡。
## What changes
- **`external-driver-floor.ts` (new):** `needsExternalDriverFloor(def)`
is true only for a non-managed datasource whose `driver` id
`resolveDriverId` does not know. `createExternalDriverFloor` is
idempotent per driver and returns a Proxy over it.
- `find`, `findOne` and `count` send `where`, `orderBy` and `limit:
1001`, never `offset` or `fields`.
- More than 1000 rows back ⇒ `ExternalRowCapExceededError`.
- Otherwise the engine applies the whole query in memory. `findOne` is
the same read with `limit: 1`; `count` is its length after `where`.
- `aggregate` is hidden (both `get` and `has`), so the engine aggregates
in memory.
- Writes pass through. A remote 4xx (read from `status`, then
`statusCode`) ⇒ `ExternalWriteRefusedError`; any other failure ⇒
`DatasourceUnavailableError` with the new reason `remote`, and the cause
is logged at warn.
- **`driver-query-in-memory.ts` (new):** `applyDriverQueryInMemory(rows,
query, { fieldClasses?, jsonStored? })` applies where, then orderBy,
then offset/limit, then fields.
- Nulls sort last in either direction.
- A paged read appends an `id` tie-break in the last key's direction, as
driver-sql does.
- `limit` is read by presence, so `limit: 0` returns no rows.
- `fields` keeps `id`, and every row returned is a copy.
- Keys the helper does not apply (`groupBy`, `aggregations`, `having`,
`search`, `searchFields`, `top`) are refused by name.
- A `ReadableIds` leaf is refused (ADR-0139 D3).
- **`having-filter.ts`:** the walker gains a `where` position
(`matchesDriverWhere`, refusal wording that names `where`). It adds
`$like` and `$ilike` through the spec's own `likePatternToRegExp`,
refusing a dangling escape, a NUL or a non-string pattern. `having` and
the per-aggregation `filter` still refuse both operators.
- **`external-driver-errors.ts` (new):** the two 422 classes, with
status declared on the thrown error (the `driver-connect-errors.ts`
pattern). `driver-connect-errors.ts` gains the per-request reason
`remote`.
- **`engine.ts`:** `DatasourceDef` gains `driver`, and
`registerDatasourceDef` stores it. The private
`applyExternalDriverFloor(name)` runs from both `registerDriver` and
`registerDatasourceDef`, because either can come first. It replaces the
map entry once, and unwraps it if a later definition no longer needs the
floor. The `registerDriver` collision check compares the unwrapped
drivers.
- **`index.ts` / `core.ts`:** the new exports (listed in the changeset).
- **Spec, separate commit:** two ledger rows under
`@objectstack/objectql` in `error-code-ledger.zod.ts`. The closed
`ApiErrorSchema.code` vocabulary gains both members, so the generated
references `contract.mdx`, `automation-api.mdx` and
`error-code-ledger.mdx` are re-recorded with `gen:schema` + `gen:docs`.
`check:api-surface` reports the surface unchanged.
`shared/external-errors.ts` is untouched, per the ruling.
-
**`packages/runtime/src/plugin-driver-external-object.spike.test.ts`:**
the P0 spike pinned the pre-floor calls, so it is re-pinned to what the
remote now sees.
- `find` and `count` both arrive as `find` with `limit: 1001`, and
`findOne` is answered from that `find`.
- `getDriverByName` returns the floored instance, which
`unwrapExternalDriverFloor` maps back to the driver.
- A `start_date` tie on the paged list now breaks by `id`.
## Tests
- `driver-query-in-memory.conformance.test.ts`: the spec tables are the
oracle. It covers FILTER_LOGIC_CASES, FILTER_TEXT_CASES (every rejection
row in the ADR-0112 envelope), PAGINATION_CASES (walked page by page),
PAGINATION_UNORDERED_CASES and PAGINATION_ZERO_LIMIT_CASES. It also
covers TEMPORAL_CASES, lowered through `lowerFilterCondition`, with
tokens resolved at TEMPORAL_NOW and rows in both writer forms. 106
passed.
- `external-driver-floor.test.ts`: runs through the real engine, with a
driver that pushes nothing down. 17 passed. It covers:
- which datasources get the floor; a built-in or managed datasource
keeps its own instance; both registration orders; a managed redefinition
unwraps;
- the one-instance pin (`getDriverByName`, `datasource()` and
`getDriver(object)` are the same object, and re-registration is
idempotent);
- the right subset under filter, sort, page and projection, and exactly
what the driver was sent;
- findOne, count, aggregate in memory, 1000 rows answered and 1001
refused on every read;
- a read throw ⇒ 503 without the cause in the message; a 4xx write ⇒ 422
with the remote message and status; a 502 write ⇒ 503.
- `driver-where-like.test.ts`: `$like` and `$ilike` semantics and
refusals, and `having` still refusing them. 6 passed.
- Suites run under the verify lock, all green:
- `@objectstack/objectql`: 482 files, 9071 tests.
- `@objectstack/rest`: 334 files.
- `@objectstack/service-datasource`: 58 files.
- `@objectstack/runtime`: 471 files, 7668 tests.
- The spike: 12/12.
- `typecheck` (tsc plus `check:test-typecheck`) is green for objectql
and runtime.
## Acceptance notes
1. **Predicate writes are not floored.** `updateMany` and `deleteMany`
pass `where` to the driver unchanged, as the ruling says (writes pass
through). A plugin driver that ignores `where` on these would write more
rows than asked. Open item for the PM: floor them as read-ids-then-by-id
writes, or refuse them on floored datasources.
2. **A `ReadableIds` leaf is refused.** A controlled-by-parent external
object, whose read scope composes a `ReadableIds` leaf, cannot be read
through the floor: it is refused per ADR-0139 D3 rather than answered
short.
3. **Every read costs a capped read.** `count` and `findOne` each read
up to 1001 rows from the remote, and a list page plus its total is two
such reads. This is the price of correctness without a pushdown
credential, which is the epic's P3.
4. **Docs are not in this PR.** #27349 has not merged, so the three
`virtual-objects.mdx` passages (fail-closed owner/sharing column, the
driver three-step progression, the 1000-row cap) and the two "planned"
sentences become a follow-up after it lands.
5. **The change is declared BREAKING.** It narrows the accept set: a
plugin-driver read that matches more than 1000 rows used to be answered
and is now 422. The changeset carries `adr-0087: not-required
(no-migration-prescription)`.
## Verification
Final head `9f12119993`.
- **Derived gates:** `node scripts/pm/dispatch-gates.mjs --commands
--repo objectstack-ai/objectstack` derives 110 families for this diff.
All 110 ran with their exit codes recorded before any pipe, and `--ran`
reconciles `110 derived, 110 run, 0 NOT-MEASURED, 0 UNRUN`.
- **Three reds on the first pass, all fixed or re-measured:**
- `check:docs`: the generated references were stale; re-recorded in the
spec commit.
- `check:query-options-erasure`: the new test's `as any` on engine query
options grew the test surface from 236 to 244; the casts were removed,
so it is back at 236.
- Two exit-3 prerequisites, which are not findings:
`check-plugin-teardown-shape --self-test` needed its pinned fixture
commit fetched, and `check:dual-build-cjs-loads` needed a built `dist`.
Both pass on re-run.
- **Full dogfood battery** (the title carries `!`): run under the verify
lock with `VERDICT command-exit 0`. Test Files 348 passed and 3 skipped
of 351; Tests 2806 passed and 20 skipped. It ran one commit before the
final head, which changed only the new test's casts and the generated
reference docs.
- **Suites, under the lock:**
- objectql: 482 files, 9071 tests.
- rest: 334 files.
- service-datasource: 58 files.
- runtime: 471 files, 7668 tests.
- spec ledger and contract tests: 93/93.
- `typecheck` for objectql and runtime, including
`check:test-typecheck`.
---
_Generated by [Claude
Code](https://claude.ai/code/session_017ihL2dgsoHnkCiSyXF8isz)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
fix(objectql)!: a plugin driver on an external datasource answers every read correctly or refuses: the engine applies the whole query to the rows the driver returns, up to 1000 of them (#27277)
7
+
8
+
Clause-②: yes (two codes join ERROR_CODE_LEDGER under `@objectstack/objectql`: `EXTERNAL_WRITE_REFUSED` and `EXTERNAL_ROW_CAP_EXCEEDED`, both 422)
9
+
Metadata-protocol: none
10
+
11
+
<!-- adr-0087: not-required (no-migration-prescription) No key of any schema is added, removed, renamed or re-shaped. A read on a plugin-driver datasource that matches more than 1000 rows is now refused with a 422 that names its own remedy (narrow the filter, or mirror the data with DataSync); there is no metadata for a migration to rewrite. -->
12
+
13
+
**BREAKING** accept-set narrowing for reads on a plugin-driver external datasource (`@objectstack/objectql`). Only a datasource with both of these is affected: `schemaMode` is not `managed`, and `driver` names a driver a server plugin published (not a built-in id: memory, sqlite, sqlite-wasm, postgres, mysql, mongodb, turso). A managed datasource and an ADR-0015 federated database on a built-in driver are unchanged.
14
+
15
+
-**Before.** The engine passed the whole query to the plugin driver and served whatever came back. A driver that ignored part of the filter, the sort or the paging answered the wrong rows, and nothing said so.
16
+
-**Now.** The engine holds such a driver behind a correctness floor. The driver is sent `where` and `orderBy`, which it may push down, and `limit: 1001`. It is never sent `offset` or `fields`. The engine then applies the whole query to the rows it gets back (`applyDriverQueryInMemory`): filter, sort (nulls last, and an `id` tie-break on a paged read), offset and limit, and projection. A driver that returned every row still answers the right subset.
17
+
-`findOne` and `count` are answered from the same read.
18
+
-`aggregate` is hidden, so the engine aggregates in memory over that read.
19
+
- The filter is evaluated by the engine's own walker (the one `having` uses). `$like` and `$ilike` now work in its `where` position. It is held to the spec tables `FILTER_LOGIC_CASES`, `FILTER_TEXT_CASES`, `PAGINATION_*_CASES` and `TEMPORAL_CASES`.
20
+
-**Row cap.** A read that matches more than 1000 rows is refused with `EXTERNAL_ROW_CAP_EXCEEDED` (422), whose message says to narrow the filter or mirror the data into a real object with DataSync.
21
+
-**Writes pass through, by id.** The floored driver does not offer `updateMany` or `deleteMany`, so a by-id write under a precondition goes through `update`, and a multi-row predicate update or delete is refused.
22
+
- A remote refusal for the request's own sake (400, 404, 409, 412 or 422) is `EXTERNAL_WRITE_REFUSED` (422). Its message is the remote's own, and `details.remoteStatus` carries the status.
23
+
- Any other failure is `ERR_DATASOURCE_UNAVAILABLE` (503, `reason: 'remote'`): no answer, a 5xx, or a 401/403/407/429 credential or rate-limit condition. The cause goes to the server log at `warn` and is never sent to the caller.
24
+
-**One instance.** The engine stores the floored driver in place of the driver, so `getDriverByName`, `datasource()` and the routing for an object all return the same instance. `unwrapExternalDriverFloor` answers the driver behind it.
25
+
-**Re-registration.** A datasource definition registered again without `driver` keeps the driver id stored before it, so the floor stays on.
26
+
-**Not covered.** A `where` with a `ReadableIds` leaf is refused (ADR-0139 D3), because the engine has no semi-join over a plugin driver's rows.
27
+
28
+
**The remedy.** If a read now meets the cap, narrow its filter, or mirror the data with a `mapping` pulled by a `job` and read the real object.
29
+
30
+
New exports from `@objectstack/objectql`: `applyDriverQueryInMemory`, `ExternalWriteRefusedError`, `ExternalRowCapExceededError`, their `*_CODE` constants, `EXTERNAL_ROW_CAP`, `EXTERNAL_WRITE_REFUSED_STATUSES`, `needsExternalDriverFloor`, `unwrapExternalDriverFloor`, and the types `DriverQueryInMemory`, `DriverQueryInMemoryOptions`, `ExternalWriteOperation` and `DatasourceUnavailableReason`. `DatasourceDef` gains an optional `driver`.
For a datasource whose `schemaMode` is not `managed` and whose `driver` is not a built-in
403
+
id (memory, sqlite, sqlite-wasm, postgres, mysql, mongodb, turso), the engine holds the
404
+
driver behind a correctness floor. It does not change a federated database on a built-in
405
+
driver.
406
+
407
+
-**What the driver is sent.**`find` gets `where` and `orderBy`, which it may push
408
+
down, and `limit: 1001`. It never gets `offset` or `fields`. `findOne` and `count`
409
+
reach your driver as that same `find`. `aggregate` is never called; the engine
410
+
aggregates in memory.
411
+
-**What the engine does with the rows.** It applies the whole query itself: `where`
412
+
(the engine's own operator vocabulary, `$like` and `$ilike` included), the sort (no
413
+
value sorts last), `offset` and `limit` (a paged read breaks ties by `id`), and the
414
+
projection.
415
+
-**Writes** go to your driver unchanged, one record at a time. The floored driver does
416
+
not offer `updateMany` or `deleteMany`, so no `where` the engine cannot re-check
417
+
reaches a write. A by-id write under a precondition goes through `update` by id, and
418
+
a multi-row predicate update or delete is refused.
419
+
420
+
So a driver grows in three steps, each correct:
421
+
422
+
1.**Return every row.** Ignore the query and answer the resource's rows. The answer is
423
+
correct for any resource with at most 1000 rows. Honour `limit` by paging the remote:
424
+
a remote whose page is smaller than 1001 rows, and that you do not page through,
425
+
answers fewer rows than match, and the engine cannot detect it.
426
+
2.**Push `where` and `orderBy` down.** The remote returns only the matching rows, so the
427
+
1000-row cap now counts matching rows, not the whole resource, and less travels. The
428
+
engine still re-checks every row, so a translation slip shows up as a missing row
429
+
(step 1's rule), never as an extra one.
430
+
3.**Skip the floor with a conformance-tested driver.** This is planned for a later
431
+
phase ([#27261](https://github.com/objectstack-ai/objectstack/issues/27261)): a driver
432
+
that passes the platform's conformance kit would be trusted with paging and the cap.
433
+
It does not exist yet.
434
+
435
+
**The 1000-row cap.** A read that matches more than 1000 rows is refused with
436
+
`422 EXTERNAL_ROW_CAP_EXCEEDED` rather than answered from a partial set. The fix is to
437
+
narrow the filter so fewer rows match, or, for data that size, to mirror it into a real
438
+
object with a [scheduled pull](/docs/automation/jobs#pulling-a-mapping) and read that.
439
+
The cap applies to plugin drivers only. It also means a list cannot page past its first
440
+
1000 matching rows: when more than 1000 rows match, every page of that read is refused,
441
+
the one at offset 1000 included. The cap is fixed and not configurable.
442
+
443
+
**Not covered by the floor.**
444
+
445
+
- Predicate writes. `updateMany` and `deleteMany` are not offered on a floored datasource,
446
+
so a write that selects its rows by a filter is refused rather than handed to your
447
+
driver. Read the ids, then write each one.
448
+
- A read whose scope composes a `ReadableIds` leaf, such as an object
449
+
`controlled_by_parent`, is refused (ADR-0139): the engine has no semi-join over a
450
+
plugin driver's rows.
451
+
377
452
### Credentials
378
453
379
454
⛔ Never put a credential in the stack — not in `config`, not in a hook body, not in an
@@ -389,16 +464,20 @@ repository.
389
464
390
465
## What your driver receives
391
466
392
-
Measured on the reference test, request by request:
467
+
Measured on the reference test, request by request, with the engine's floor
468
+
([above](#the-engine-re-applies-every-query)) in place:
393
469
394
470
| The platform does | Your driver is called with |
395
471
| :--- | :--- |
396
472
| Boots |`connect()` once. No `syncSchema`, no `introspectSchema` (`checkOnBoot: false`). |
397
-
| Serves a list (`GET /data/berth_slot` with the view's filter, sort and page size) |`find(object, { where, orderBy, limit})`, then `count(object, { where})` for the total |
398
-
| Serves a record (`GET /data/berth_slot/:id`) |`findOne(object, { where: { id }, limit: 1 })`|
399
-
| Writes (`PATCH /data/berth_slot/:id`) | several by-id reads (the pre-image for hooks and checks), then `update(object, id, patch)` with the extra keys above |
473
+
| Serves a list (`GET /data/berth_slot` with the view's filter, sort and page size) |`find(object, { where, orderBy, limit: 1001 })`, then `find(object, { where, limit: 1001 })` for the total. The page size is applied by the engine.|
474
+
| Serves a record (`GET /data/berth_slot/:id`) |`find(object, { where: { id }, limit: 1001 })`|
475
+
| Writes (`PATCH /data/berth_slot/:id`) | several by-id reads as `find`(the pre-image for hooks and checks), then `update(object, id, patch)` with the extra keys above |
400
476
| Refuses a caller with no grant on a list read | nothing |
401
477
478
+
Your driver should still implement `findOne` and `count`: the engine calls them on a
479
+
driver that is not behind the floor, and the shape test below exercises them.
480
+
402
481
Two details catch every first driver:
403
482
404
483
-**`object` is the object name** (`berth_slot`), never `external.remoteName` (`slots`).
Copy file name to clipboardExpand all lines: content/docs/references/api/automation-api.mdx
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -347,7 +347,7 @@ const result = AutomationApiErrorCode.parse(data);
347
347
348
348
| Property | Type | Required | Description |
349
349
| :--- | :--- | :--- | :--- |
350
-
|**code**|`Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +347 more>`| ✅ | The inner refusal's code in its WIRE spelling - the ADR-0112 member a record save of the same write answers in `error.code` (e.g. `UNIQUE_VIOLATION`); the closed set `ApiErrorSchema.code` parses against |
350
+
|**code**|`Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +349 more>`| ✅ | The inner refusal's code in its WIRE spelling - the ADR-0112 member a record save of the same write answers in `error.code` (e.g. `UNIQUE_VIOLATION`); the closed set `ApiErrorSchema.code` parses against |
351
351
|**message**|`string`| ✅ | The inner refusal's own diagnostic text, verbatim, without the node and operation prefixes the run's failure text adds |
352
352
353
353
### Allowed Values: `FlowInnerRefusal.code`
@@ -496,9 +496,11 @@ const result = AutomationApiErrorCode.parse(data);
Copy file name to clipboardExpand all lines: content/docs/references/api/contract.mdx
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -28,7 +28,7 @@ const result = ApiErrorSchema.parse(data);
28
28
29
29
| Property | Type | Required | Description |
30
30
| :--- | :--- | :--- | :--- |
31
-
|**code**|`Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +347 more>`| ✅ | Error code (e.g. VALIDATION_ERROR; StandardErrorCode ∪ the ledger the serving side registers — ERROR_CODE_LEDGER for framework packages) |
31
+
|**code**|`Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +349 more>`| ✅ | Error code (e.g. VALIDATION_ERROR; StandardErrorCode ∪ the ledger the serving side registers — ERROR_CODE_LEDGER for framework packages) |
32
32
|**declaredCode**|`string`| optional | The producer-declared code, verbatim, when it is not a member of the closed `code` vocabulary — the open, author-authored channel (app-specific spellings; ADR-0112) |
|**userMessage**|`string`| optional | Producer-marked user-facing refusal text, verbatim. Present exactly when the producer opted in at throw time; consumers render it to end users and keep their generic substitution for anything unmarked. Status-agnostic; never replaces `message`. |
@@ -184,9 +184,11 @@ const result = ApiErrorSchema.parse(data);
0 commit comments