feat: add generate-spec command#2937
Conversation
🦋 Changeset detectedLatest commit: ad8c15f The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
2c94a80 to
6b62b28
Compare
sobanieca-redocly
changed the title
generate-spec commandgenerate-spec command
Performance Benchmark (Lower is Faster)
|
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
@cursor review |
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
2 issues from previous reviews remain unresolved.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 956bf00. Configure here.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 3a11b20. Configure here.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit d7ee556. Configure here.
|
@cursor review |
| error instanceof Error ? error.message : String(error) | ||
| }\n` | ||
| ); | ||
| } |
There was a problem hiding this comment.
Final lint drops refinements
Medium Severity
When --with-ai runs, refineSpecWithAi can accept and merge some operations, then fail on the final whole-document lint. The handler catches that error and writes the original baseline YAML, so successful per-operation refinements are discarded instead of being emitted.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 896fd97. Configure here.
There was a problem hiding this comment.
Intentional. If the final whole-document lint fails, the merged document is invalid, and emitting an invalid description would be worse than falling back to the deterministic baseline — the only output we can guarantee. Each fragment already passed lint against the full component set at merge time, so a final-lint failure can only come from concurrent merge interactions; the warning tells the user why the fallback happened.
Co-authored-by: Jacek Łękawa <164185257+JLekawa@users.noreply.github.com>
| }); | ||
| }); | ||
|
|
||
| describe('generate-spec - AI refinement', () => { |
There was a problem hiding this comment.
can we test the prompt somehow? to avoid regression while editing
There was a problem hiding this comment.
Added __tests__/prompt.test.ts — it snapshots the full system instructions and a built user prompt, so any prompt edit produces a reviewable snapshot diff (it already caught the 3.1 → 3.2 wording change).
There was a problem hiding this comment.
i was thinking more about a prompt quality check - eval harness or some kind of .md file with test cases
eval harness seems like overkill for a first iteration, but at least the test cases you leaned on when testing the agents' responses after the modification would be good to have
| } | ||
| return lines.join('\n'); | ||
| }) | ||
| .join('\n\n'); |
There was a problem hiding this comment.
Traffic content enables prompt injection
Medium Severity
--with-ai embeds recorded request and response bodies (and inferred operation YAML) directly into the user prompt with no structural delimiters or escaping. Malicious or crafted traffic can inject instructions that compete with the fixed system rules, affecting refinement output and potentially weakening safeguards described in the prompt.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit e9e957f. Configure here.
There was a problem hiding this comment.
True in principle, but delimiters or escaping can't prevent injection — the model reads the text either way. The defense is downstream: providers run with tools disabled or in a read-only sandbox from an empty temp directory, and a response is only merged if it keeps the path/method/statuses/operationId, parses as the expected fragment, does not redefine reserved components, and passes the spec lint. An injected instruction can at most corrupt the refinement of its own operation, which those checks reject back to the deterministic baseline. The README also warns that --with-ai sends traffic samples to the provider.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4365a83. Configure here.
| paths: { [options.path]: { [options.method]: fragment.operation } }, | ||
| components: { schemas: { ...schemas, ...fragment.components } }, | ||
| }; | ||
| await lintDocumentSource(stringifyYaml(fragmentDocument), options.config); |
There was a problem hiding this comment.
Stale snapshot breaks parallel AI merge
Medium Severity
With default --ai-concurrency above 1, each refineOperation captures components.schemas and reserved names before the provider call. After other workers merge new or updated components, acceptance still uses that stale snapshot, so reserved-component rejection and per-operation lint can disagree with the live document and Object.assign can overwrite fresher shared schemas.
Reviewed by Cursor Bugbot for commit 4365a83. Configure here.
There was a problem hiding this comment.
Mostly a restatement of the documented concurrency trade-off, with one correction: the per-operation lint isn't stale — schemas is a live reference and the fragment document is built after the provider call, so lint sees all merges up to that point. The reserved-name list is intentionally fixed at prompt time because it must match what the model was told it may not touch. What remains is the documented last-merge-wins rule for shared components: merges are serialized on the event loop, the final whole-document lint guards the merged result, and --ai-concurrency 1 gives strictly sequential behavior. Making acceptance transactional would add locking/re-validation for an edge the final lint already covers.
| - Identifier-like path segments (numeric, UUID, ULID, CUID, prefixed and opaque tokens) become named path parameters. | ||
| - Body schemas are merged across all observations. | ||
| A property becomes optional as soon as one sample omits it. | ||
| - Alternative body shapes for the same operation are preserved as `oneOf` variants instead of being collapsed, and values observed as `null` produce type unions such as `["string", "null"]`. |
There was a problem hiding this comment.
| - Alternative body shapes for the same operation are preserved as `oneOf` variants instead of being collapsed, and values observed as `null` produce type unions such as `["string", "null"]`. | |
| - Alternative body shapes for the same operation are preserved as `oneOf` variants instead of being collapsed. | |
| - Values observed as `null` produce type unions such as `["string", "null"]`. |
| The same entity is recognized when it appears as a list item and as a single resource, with different `required` sets, or with near-identical properties (at least 75% shared, with compatible types). | ||
| Components are named from the path entity, the enclosing property name, or `Error` for error responses. | ||
| A shape repeated only because its parent shape repeats stays inline. |
There was a problem hiding this comment.
| The same entity is recognized when it appears as a list item and as a single resource, with different `required` sets, or with near-identical properties (at least 75% shared, with compatible types). | |
| Components are named from the path entity, the enclosing property name, or `Error` for error responses. | |
| A shape repeated only because its parent shape repeats stays inline. | |
| - The same entity is recognized when it appears as a list item and as a single resource, with different `required` sets, or with near-identical properties (at least 75% shared, with compatible types). | |
| - Components are named from the path entity, the enclosing property name, or `Error` for error responses. | |
| - A shape repeated only because its parent shape repeats stays inline. |
| When a string only ever takes a small set of identifier-like values with enough repetition (at least 4 observations, at most 5 distinct values, each seen twice on average) it becomes an `enum`. | ||
| Enums apply to body properties and query parameters. | ||
| Path parameters and nullable unions get formats only. | ||
| Evidence is pooled across all operations a shared component was observed in. |
There was a problem hiding this comment.
| When a string only ever takes a small set of identifier-like values with enough repetition (at least 4 observations, at most 5 distinct values, each seen twice on average) it becomes an `enum`. | |
| Enums apply to body properties and query parameters. | |
| Path parameters and nullable unions get formats only. | |
| Evidence is pooled across all operations a shared component was observed in. | |
| - When a string only ever takes a small set of identifier-like values with enough repetition it becomes an `enum`. | |
| To trigger this conversion, the command needs at least 4 observations, at most 5 distinct values, each seen twice on average. | |
| - Enums apply to body properties and query parameters. | |
| - Path parameters and nullable unions get formats only. | |
| - Evidence is pooled across all operations a shared component was observed in. |
|
|
||
| ## AI refinement | ||
|
|
||
| The description is refined one operation at a time, so prompt and response stay small no matter how large the recorded traffic or the resulting description is. |
There was a problem hiding this comment.
| The description is refined one operation at a time, so prompt and response stay small no matter how large the recorded traffic or the resulting description is. | |
| The description is refined one operation at a time. | |
| The prompt and response stay small regardless of the volume of recorded traffic or the resulting description's size. |
| Each prompt carries a single operation from the baseline, the component schemas it references, the names of the other components (reserved against collisions), and a capped, shape-diverse sample of the real exchanges recorded for that operation (grouped by status and body shape, selected round-robin so every observed payload variant is represented). | ||
| The AI is instructed to narrow types, add formats, enums, descriptions and examples, refine or add `components/schemas`, and model alternative payloads explicitly with `oneOf` (plus `discriminator`) and `allOf` composition. |
There was a problem hiding this comment.
| Each prompt carries a single operation from the baseline, the component schemas it references, the names of the other components (reserved against collisions), and a capped, shape-diverse sample of the real exchanges recorded for that operation (grouped by status and body shape, selected round-robin so every observed payload variant is represented). | |
| The AI is instructed to narrow types, add formats, enums, descriptions and examples, refine or add `components/schemas`, and model alternative payloads explicitly with `oneOf` (plus `discriminator`) and `allOf` composition. | |
| Each prompt carries: | |
| - a single operation from the baseline | |
| - referenced component schemas | |
| - names of the other components (reserved against collisions) | |
| - a capped, shape-diverse sample of real exchanges recorded for that operation | |
| The sample of exchanges is grouped by status and body shape, selected round-robin to represent every observed payload variant. | |
| The AI: | |
| - narrows types | |
| - adds formats, enums, descriptions and examples | |
| - refines or adds `components/schemas` | |
| - models alternative payloads explicitly with `oneOf` (plus `discriminator`) and `allOf` composition |
There was a problem hiding this comment.
Applied via your later split suggestions for these paragraphs (the "Each prompt carries:" and "The AI is instructed to:" lists) in dcb7ff6.
| > **Warning:** `--with-ai` sends samples of the recorded traffic (URLs, query strings, request and response bodies) to the selected AI provider. | ||
| > Make sure the traffic contains no secrets or personal data you are not allowed to share. |
There was a problem hiding this comment.
Should this be wrapped in a Markdoc admonition tag?
There was a problem hiding this comment.
This README ships with the package on GitHub/npm rather than the docs site, so a Markdoc admonition tag would render as literal text there. Keeping the plain blockquote for now — when the command graduates from experimental and gets a docs/@v2 page, that page should use the admonition tag.
| ## AI refinement | ||
|
|
||
| The description is refined one operation at a time, so prompt and response stay small no matter how large the recorded traffic or the resulting description is. | ||
| Each prompt carries a single operation from the baseline, the component schemas it references, the names of the other components (reserved against collisions), and a capped, shape-diverse sample of the real exchanges recorded for that operation (grouped by status and body shape, selected round-robin so every observed payload variant is represented). |
There was a problem hiding this comment.
| Each prompt carries a single operation from the baseline, the component schemas it references, the names of the other components (reserved against collisions), and a capped, shape-diverse sample of the real exchanges recorded for that operation (grouped by status and body shape, selected round-robin so every observed payload variant is represented). | |
| Each prompt carries: | |
| - a single operation from the baseline | |
| - component schemas it references | |
| - names of other components (reserved against collisions) | |
| - a capped, shape-diverse sample of the real exchanges recorded for that operation | |
| The sample of exchanges is grouped by status and body shape, selected round-robin to represent every observed payload variant. | |
|
|
||
| The description is refined one operation at a time, so prompt and response stay small no matter how large the recorded traffic or the resulting description is. | ||
| Each prompt carries a single operation from the baseline, the component schemas it references, the names of the other components (reserved against collisions), and a capped, shape-diverse sample of the real exchanges recorded for that operation (grouped by status and body shape, selected round-robin so every observed payload variant is represented). | ||
| The AI is instructed to narrow types, add formats, enums, descriptions and examples, refine or add `components/schemas`, and model alternative payloads explicitly with `oneOf` (plus `discriminator`) and `allOf` composition. |
There was a problem hiding this comment.
| The AI is instructed to narrow types, add formats, enums, descriptions and examples, refine or add `components/schemas`, and model alternative payloads explicitly with `oneOf` (plus `discriminator`) and `allOf` composition. | |
| The AI is instructed to: | |
| - narrow types | |
| - add formats, enums, descriptions and examples | |
| - refine or add `components/schemas` | |
| - model alternative payloads explicitly with `oneOf` (plus `discriminator`) and `allOf` composition. |
| Each prompt carries a single operation from the baseline, the component schemas it references, the names of the other components (reserved against collisions), and a capped, shape-diverse sample of the real exchanges recorded for that operation (grouped by status and body shape, selected round-robin so every observed payload variant is represented). | ||
| The AI is instructed to narrow types, add formats, enums, descriptions and examples, refine or add `components/schemas`, and model alternative payloads explicitly with `oneOf` (plus `discriminator`) and `allOf` composition. | ||
|
|
||
| Up to `--ai-concurrency` operations (default 4) are refined in parallel, and every accepted refinement is merged back as it arrives, so operations prompted later see already-refined shared components. |
There was a problem hiding this comment.
| Up to `--ai-concurrency` operations (default 4) are refined in parallel, and every accepted refinement is merged back as it arrives, so operations prompted later see already-refined shared components. | |
| You can set the number of operations refined in parallel using `--ai-concurrency`. | |
| The default value is `4`. | |
| Every accepted refinement is merged back as it arrives: operations prompted later see already-refined shared components. |
| - does not redefine reserved components (those used only by other operations) | ||
| - passes validation with the `spec` ruleset (checked against the description's full component set) | ||
|
|
||
| An operation whose refinement is rejected keeps its deterministic baseline (reported with the reason), components no operation references anymore are pruned, and the final document is linted again as a whole. |
There was a problem hiding this comment.
| An operation whose refinement is rejected keeps its deterministic baseline (reported with the reason), components no operation references anymore are pruned, and the final document is linted again as a whole. | |
| An operation whose refinement is rejected keeps its deterministic baseline and the reason for rejection is reported). | |
| Components that no operation references are pruned. | |
| The final document is linted again as a whole. | |
There was a problem hiding this comment.
Applied in dcb7ff6 (dropped the stray closing parenthesis after "reported"), thanks.
vadyvas
left a comment
vadyvas
left a comment
There was a problem hiding this comment.
left a few comments, but overall it looks good, thanks 🔥
4 participants
What/Why/How?
Added new
generate-speccommand which will infer OAD basing on traffic data. To improve accuracy, there is option to use AI agents to improve results.Reference
Testing
Screenshots (optional)
Check yourself
Security
Note
Medium Risk
Large new surface area and
--with-aisends traffic samples to external CLIs, but refinement is gated by validation and AI failure falls back to the deterministic baseline without failing the command.Overview
Introduces an experimental
generate-specCLI command that reads HAR/Kong/Nginx/Apache JSON/NDJSON traffic (same parsers asdrift) and emits an OpenAPI 3.2 document.Deterministic pipeline: path templating for ID-like segments, merged request/response schemas (
oneOf, nullable unions), repeated shapes lifted tocomponents/schemas, and conservative string format/enum inference. Flags include--server,--title, and-o.Optional
--with-ai: refines one operation at a time via localclaude,codex, orcursorCLIs (isolated temp cwd, configurable concurrency). Accepts only refinements that preserve path/method/status codes and pass the built-inspecruleset; otherwise keeps the baseline and falls back on total failure.Also adds v2 docs, a minor changeset, CODEOWNERS, broad unit tests, and e2e coverage with stub AI binaries.
Reviewed by Cursor Bugbot for commit ad8c15f. Bugbot is set up for automated code reviews on this repo. Configure here.