Skip to content

feat(connector-openapi): an offline normalize-openapi script — Swagger 2.0 to 3.0, YAML to JSON, $ref bundled and inlined, absolute server, sha256 and an include list - #27436

Merged
objectstack-fleet[bot] merged 4 commits into
mainfrom
claude/issue-27373-openapi-normalize
Oct 29, 2026
Merged

objectstack-fleet[bot] merged 4 commits into
mainfrom
claude/issue-27373-openapi-normalize

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #27373

Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:connector-openapi 包内新增离线脚本 normalize-openapi(只在仓库里,不进 npm 包),把厂商的 Swagger 2.0 / YAML / 多文件 OpenAPI 文档转成 openapi provider 直接能加载的单个 3.x JSON,并打印写入字节的 sha256 和按模式生成的 include 列表。
  • 为什么改:上架流水线和应用作者拿到的厂商文档多数是 2.0、YAML 或满是 $ref,运行时按设计拒收 2.0、并静默丢掉 $ref 形式的参数/请求体/响应;没有这一步,每个作者都得手工转换,而且会悄悄转错。
  • 风险与回滚:运行时零改动(src/ 与发布的 dist/ 字节不变),只新增两个 devDependency(Scalar 的 @scalar/json-magic、@scalar/openapi-upgrader,均 MIT);回滚即 revert 本 PR。
  • 先看:① scripts/normalize-openapi.mts 的 Dereferencer.follow / stub(在递归处与 --max-depth 处把 schema 引用换成带说明的类型桩,这是有损的取舍);② fillEmptyConsumes(空 consumes 默认拒收);③ package.json 的依赖与 typecheck 行。
  • 两分钟验证:pnpm --filter @objectstack/connector-openapi exec vitest run scripts/normalize-openapi.test.ts,预期 Tests 13 passed (13)。
  • 需要你做的:无,席位落地。

What changes

  • packages/connectors/connector-openapi/scripts/normalize-openapi.mts (new): one script, a small CLI and an exported normalizeOpenApi():
    pnpm --filter @objectstack/connector-openapi normalize-openapi --in FILE_OR_URL --out FILE.json [--include-pattern REGEX]... [--max-depth N] [--server-url URL] [--empty-consumes MEDIA_TYPE].
    1. reads a file or an http(s) URL, JSON or YAML;
    2. bundles every external $ref (sibling files of a file document, confined to its directory; URLs only for a URL document);
    3. converts Swagger 2.0 to OpenAPI 3.0; a 3.x document keeps its version;
    4. inlines every $ref reachable from paths / webhooks, then drops the components sections only references used (securitySchemes stay, they are referenced by name);
    5. makes servers[0].url absolute (server variables take their defaults; a relative URL resolves against the source URL, or --server-url; a relative URL in a file document is refused until --server-url is given);
    6. with --include-pattern, emits the providerConfig.include list: every operation whose action key, or whose METHOD /path, matches. The key is computed with the provider's own operationActionKey (imported from src/openapi-connector.ts, not re-derived), so the list and the provider can never disagree;
    7. writes JSON.stringify(doc, null, 2) plus a newline and prints a JSON summary whose sha256 is the hash of exactly those bytes, with the cut counts and refsLeft.
      Exit codes: 0 written, 1 refused (message names the remedy), 2 usage.
  • Where inlining stops, and why it has to. "No $ref left" and "every schema fully inlined" cannot both hold for Stripe: measured on spec3.json (3.0.0, 431 paths, 612 operations, 117 back-edges among component schemas), every one of the 612 operations expands past 2,000,000 JSON nodes when only true recursion is cut. So a SCHEMA reference becomes a stub when it is recursive, or deeper than --max-depth named-schema hops (default 2); a $ref inside a vendor x- extension value is stubbed too, since its target is dropped with components. The stub keeps the target's type, nullable, title, description and says which reference it replaced and why. Structural references (parameters, responses, request bodies, headers, path items) are always inlined. Literal data (example, a schema's default / enum / const / examples, Example Objects) is copied, never walked; a PROPERTY named default is still a schema.
  • The one Swagger 2.0 repair, refused by default. An operation-level consumes: [] clears the global list, so a faithful conversion gives that request body no media type and therefore no schema: the body would vanish from the action silently. DocuSign's eSignature document does this on 187 of its 414 operations. The script refuses and names the operations; --empty-consumes application/json fills them, and the summary counts them.
  • packages/connectors/connector-openapi/tsconfig.scripts.json (new) puts scripts/** in a tsc program, named by the package's typecheck script (precedent packages/spec/tsconfig.scripts.json); without it the new directory would be TypeScript no program reads (check:type-check-coverage SOURCES_COVERED / TESTS_COVERED). This file is outside the claim's declared file surface (one package-root config); it is the only such path.
  • package.json: normalize-openapi script entry, the typecheck line, two devDependencies. pnpm-lock.yaml: regenerated by pnpm install --lockfile-only (38 added lines, nothing else moved).
  • content/docs/automation/connectors.mdx: one paragraph and one command where the page says "convert it to OpenAPI 3.x offline".
  • ⛔ openapi-provider.ts and openapi-connector.ts are untouched; the provider still refuses every non-3.x document.

The new third-party dependencies (read from the npm registry 2026-10-29)

Package Role Licence Version Last release Weekly downloads Direct deps
@scalar/openapi-upgrader Swagger 2.0 → OpenAPI 3.0 (upgrade(doc, '3.0')) MIT 0.4.6 2026-10-23 5,508,440 (order 10^6) 2 (@scalar/helpers, @scalar/openapi-types)
@scalar/json-magic bundle external $ref (files, URLs, YAML/JSON parse) MIT 0.17.0 2026-10-23 7,140,252 (order 10^6) 4 (@scalar/helpers, pathe, undici, yaml)

Both live in the scalar/scalar monorepo and release roughly weekly (38 and 74 published versions). Both are pre-1.0, so the caret ranges pin the minor. The lockfile adds four packages in all (@scalar/helpers 0.16.4 and @scalar/openapi-types 0.9.8 transitively; pathe, undici, yaml were already in the tree). Weekly counts come from the registry search endpoint (registry.npmjs.org/-/v1/search), because api.npmjs.org is refused by this container's egress proxy.

Why two packages and not one: @scalar/openapi-parser (MIT, 0.29.18, 2026-10-28) would be the one-package option, but its own upgrade() targets 3.1 only (the card's scope is 2.0 → 3.0), its load() is deprecated in favour of @scalar/json-magic/bundle, and it depends on both of these packages plus a validator (ajv). So the two direct dependencies are a strictly smaller tree. Alternatives read and rejected: swagger2openapi 7.0.8 (BSD-3-Clause, last release 2021-07-07), @apidevtools/swagger-parser 13.1.0 (MIT; parses and dereferences, does not convert 2.0), @redocly/openapi-core 2.72.0 (MIT; bundles, does not convert 2.0, 13 direct deps). The inlining itself is this script's own code: no library inlines a recursive document into finite JSON, which is the cut policy above.

Tests

scripts/normalize-openapi.test.ts, 13 cases, no network (a URL input is served by an injected fetch). Every normalized output is loaded through the REAL provider (createOpenApiProviderFactory, file-path spec via loadPackageFile), and "no $ref" is asserted on both the document and the materialized connector definition.

  • Fixtures (hand-written, trimmed, not copies of the vendor files): scripts/fixtures/swagger2-esignature-shape.json (Swagger 2.0, host / basePath, body parameter, an operation-level consumes: [], a recursive definition) and scripts/fixtures/openapi3-billing-shape.yaml plus billing-shared.yaml (OpenAPI 3.0 YAML across two files, customer ↔ subscription recursion, a vendor x-expansionResources extension holding $refs, shared responses.default and parameter, a property named default).
  • Pinned: the 2.0 refusal names the operation; conversion keeps the body schema and path parameter in the action; recursive stub; external file bundled; structural refs inlined; default as property vs literal; extension stub; action keys equal the provider's; the generated include list materializes exactly those actions; zero-match and invalid patterns refused; --max-depth 0 stubs; relative server resolved against a source URL, variables filled, refused from a file without --server-url; non-OpenAPI and HTTP 404 refused; the CLI's printed sha256 equals the hash of the file it wrote; exit codes 1 and 2 write nothing.
  • One-time ablation (not kept): removing the inSchema && guard on the schema-literal keys (so responses.default is read as literal data) turned 2 of 13 red ("bundles the external file…", "inlines up to --max-depth…"); scripts/ablation-replace.mjs reported anchor 1 → 0, blob 613cae347a0e → 0cd0bc79a1fd, restored to the HEAD blob with git diff HEAD empty.

Acceptance notes

  • Acceptance 1, real documents, run once locally (not in CI, no network in tests). Stripe stripe/openapi openapi/spec3.json (8,317,513 bytes, 3.0.0) → 63,317,655 bytes, refsLeft 0, cuts 754 recursive / 29,831 max-depth / 8,164 extension; the provider materializes 612 actions with 0 $ref in the definition. The same document by URL with --include-pattern '^GET /v1/customers' gives the byte-identical output (same sha256) and an include list of 22 keys; the provider with that include materializes exactly 22 actions. DocuSign docusign/OpenAPI-Specifications esignature.rest.swagger-v2.1.json (4,846,375 bytes, Swagger 2.0) is refused without --empty-consumes (187 operations named); with --empty-consumes application/json → OpenAPI 3.0.4, 52,884,345 bytes, refsLeft 0, server https://www.docusign.net/restapi; the provider materializes 414 actions, 0 $ref, 192 with a body input.
  • Acceptance 2: the printed sha256 equals sha256sum of the written file for both real outputs (Stripe 0ea3c37a…b2, DocuSign ceee5238…37), and the CLI test pins it.
  • Acceptance 3: no changeset. The script does not ship: files is dist, README.md, CHANGELOG.md; README and CHANGELOG are untouched; after pnpm --filter @objectstack/connector-openapi build, normalizeOpenApi / normalize-openapi occur 0 times in dist/* while the positive control createOpenApiProviderFactory occurs 4 times in dist/index.js. Labelled skip-changeset.
  • Size vs --max-depth (whole document, no include): depth 0 → Stripe 6.1 MB / DocuSign 1.6 MB; 1 → 14.8 / 4.9 MB; 2 (default) → 63.3 / 52.9 MB; 3 → 361 / 160 MB. The default is a trade-off the reviewer may want to change; it is one constant (DEFAULT_MAX_DEPTH).
  • Reachability: the script is reachable only from a checkout of this repository (it is not in the npm tarball and has no bin). A pipeline in another repository cannot run it from npm; publishing it (a bin, the two packages moved to dependencies) would widen the published surface and is left to a ruling.
  • Not done here, by scope: a 3.x document's path-item-level parameters are not hoisted into operations (the 2.0 upgrader hoists them for 2.0 input). The generator does not read path-item-level parameters at all, so such a parameter is missing from the action's inputSchema; that is a runtime defect, reported separately rather than worked around here.
  • discriminator.mapping values are strings, not $ref keys; after components is dropped they name schemas no longer in the document. The generator never reads discriminator.
  • Upstream: @scalar/openapi-upgrader lets the global produces win over an operation-level one (document.produces ?? operationItem.produces); harmless for both real documents, noted only.

Verification

All local readings at head f9a4f408.

Package test and typecheck (head f9a4f40)
$ bash scripts/pm/os-verify-lock.sh -c "pnpm --filter @objectstack/connector-openapi test && pnpm --filter @objectstack/connector-openapi typecheck"
 Test Files  5 passed (5)
      Tests  63 passed (63)
> tsc --noEmit && tsc --noEmit -p tsconfig.scripts.json && pnpm check:test-typecheck
check:test-typecheck: OK — @objectstack/connector-openapi's test layer compiles under packages/connectors/connector-openapi/tsconfig.test.json
os-verify-lock: VERDICT command-exit 0
$ tsc --noEmit -p tsconfig.scripts.json --listFiles | grep connector-openapi/scripts/
.../scripts/normalize-openapi.mts
.../scripts/normalize-openapi.test.ts
Derived gate families

node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 89 commands for this diff; each was run with its exit code captured before any pipe. 86 exited 0. 3 exited 3 (PREREQUISITE NOT MET, nothing measured, none reads these paths): check-plugin-teardown-shape.mjs --self-test (shallow clone), @objectstack/spec check:skill-examples (client packages unbuilt), check:dual-build-cjs-loads (workspace unbuilt). --ran reconciliation: "89 derived famil(ies) accounted for — 86 run, 3 NOT-MEASURED (3 DERIVED from a recorded exit 3)". The full rerun at the final head is in the report comment on #27373.

Real-document runs (local, once)
$ pnpm -s normalize-openapi --in stripe-spec3.json --out stripe.json
{ "bytes": 63317655, "sha256": "0ea3c37a0854ad2646a156404b1ef86cca57403855e3803cd4630601eb6fc7b2",
  "sourceVersion": "3.0.0", "openapi": "3.0.0", "operations": 612, "maxDepth": 2,
  "cuts": { "recursive": 754, "maxDepth": 29831, "extension": 8164 }, "refsLeft": 0, "filledConsumes": 0 }
provider: actions 612, handlers 612, $ref in def 0

$ pnpm -s normalize-openapi --in esignature.rest.swagger-v2.1.json --out docusign.json
normalize-openapi: 187 Swagger 2.0 operation(s) take a request body but declare an empty 'consumes' list ... (exit 1)
$ pnpm -s normalize-openapi --in esignature.rest.swagger-v2.1.json --out docusign.json --empty-consumes application/json
{ "bytes": 52884345, "sha256": "ceee523822c266fc1de6e5433915f6e4385753d2b239ce8b94f48cafe7d93d37",
  "sourceVersion": "2.0", "openapi": "3.0.4", "operations": 414,
  "cuts": { "recursive": 30, "maxDepth": 68434, "extension": 0 }, "refsLeft": 0, "filledConsumes": 187 }
provider: actions 414, handlers 414, $ref in def 0, actions with body input 192

$ sha256sum docusign.json stripe.json
ceee523822c266fc1de6e5433915f6e4385753d2b239ce8b94f48cafe7d93d37  docusign.json
0ea3c37a0854ad2646a156404b1ef86cca57403855e3803cd4630601eb6fc7b2  stripe.json

Generated by Claude Code

…ependencies for the offline normalizer

Claude-Session: https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U
Co-authored-by: Claude <noreply@anthropic.com>
…er 2.0 to 3.0, YAML to JSON, bundled and dereferenced, absolute server, sha256 and an include list

Claude-Session: https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U
Co-authored-by: Claude <noreply@anthropic.com>
…ize-openapi script

Claude-Session: https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet objectstack-fleet Bot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Oct 29, 2026
@github-actions github-actions Bot added size/xl dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation tests labels Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

⚠️ 1 changed file(s) yielded no anchor (packages/connectors/connector-openapi/tsconfig.scripts.json), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files. Nothing else in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 1 changed package(s)).

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/connectors/connector-openapi/tsconfig.scripts.json) — pages documenting those are invisible to this run
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 2 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 3bfb357aa37e043420bcd08ab45d5a7f5335cb86 → packageMentionDocs.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

epic:#27293#1 · sunhaolin · session_017xRBcVBKAFk1ntXSjvPt7U

席位核对

ACCEPT at head f9a4f408 · Implemented-by: claude/issue-27373-openapi-normalize · Reviewed-by: this seat. No contract-review record is owed: Clause-②: no, no packages/spec/src/** path, no governed path (check-governed-merges --pr 27436: NOT governed), so CI plus this seat's read is the review.

Check Result
Shape draft, base main, first line Fixes #27373; the only closing keyword
Scope 9 files, +1228/−1; runtime src/ untouched; the script is outside files (dist, README.md, CHANGELOG.md), so nothing published changes and skip-changeset is right. Surface addendum: tsconfig.scripts.json (without it no tsc program reads scripts/**; precedent packages/spec/tsconfig.scripts.json)
Dependencies @scalar/openapi-upgrader 0.4.6 and @scalar/json-magic 0.17.0, both devDependencies, both MIT, both latest published 2026-10-23 — re-read from registry.npmjs.org by this seat, matching the PR's table. Lockfile regenerated with pnpm install --lockfile-only (+38 lines, no unrelated peer flip)
Gates 71 check-runs on the head: 59 success, 12 skipped (all on main's roster, check-expected-skips OK); mergeable clean
Evidence 13 script tests, each loading its output through the real createOpenApiProviderFactory with no $ref; the real Stripe (612 actions) and DocuSign (414 actions) documents normalized once locally with the sha256 matching sha256sum
Docs, checked line by line connectors.mdx: "not shipped in the npm package" (true by files), the CLI flags (--in, --out, --include-pattern, --max-depth default 2, --empty-consumes) as the script defines them, and "the runtime never converts" (unchanged)

The dev's three open questions, decided by the seat (implementation parameters, inside triage's ruling 6487372008; no product or contract fork):

  1. Reachability — A: stays a repo script, as triage ruled. The cloud shelf pipeline runs it from an objectstack checkout; whether it can is the cloud side's fact, carried by E4's pipeline card ([Decision] E4 of #27293 — official connector apps: which are open edition (this repository) and which commercial (cloud repository), and when the first shelf starts #27302's ruling files it in the cloud repository).
  2. Inlining depth — A: default --max-depth 2, a stub naming each replaced reference; --include-pattern keeps real shelf documents small.
  3. Empty consumes — A: refused by default, named per operation; --empty-consumes is the explicit opt-in. A guessed media type is the silent-wrong class the script exists to remove.

Out of scope: filed #27463 (path-item parameters never read by the generator, class a) · Acceptance notes: discriminator.mapping names dropped components (dormant), the upgrader's produces precedence (upstream, no effect on either real document).

Landing: not governed: the seat lands it now (pr_ready + automerge_enable).

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review October 29, 2026 07:14
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge October 29, 2026 07:14
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Oct 29, 2026
Merged via the queue into main with commit 262ab67 Oct 29, 2026
72 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-27373-openapi-normalize branch October 29, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation size/xl skip-changeset PR has no user-facing published change; bypasses the changeset gate tests

Projects

None yet

1 participant