Repository navigation
Rebuild events stream for subscribe on paused turns - #1165
Conversation
🦋 Changeset detectedLatest commit: 455dfb1 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # packages/trueforge/src/runtime/activeTurns.ts
Stream migrations must run after the pause migrations dated 20261028. Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve storeContractSuite import conflict by keeping TerminalTurnState (stream rebuild) and TurnNonTerminalMetrics (pause emit), and adapt HEAD-only last_turn_details tests to the pause-branch metrics fields. Co-authored-by: Cursor <cursoragent@cursor.com>
Running and finished turns keep the existing stream or fall back to list events, and subscribe does not rebuild the turn. Co-authored-by: Cursor <cursoragent@cursor.com>
…ry/trueforge into cj-turn-stream-rebuild
* Subscribe and repair turn streams by event id.
New live and repaired streams use Redis ids {ulidMillisecond}-{n} and do not assign sequence numbers. Subscribe accepts from_event_id or after_event_id. after_sequence_number remains only for older integer streams.
* Fix event-id resume on legacy streams, repair, and reconnect.
Scan leftover sequence streams from the start, retry a distinct shared-id payload after a timestamp conflict, and match repair rows without depending on JSON key order. Replay mcp.auth_required, and do not treat that inclusive shell as reconnect progress.
* Minor fixes from self review
* Regenerate OpenAPI document and SDKs
* Simplify redis lua scripts
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: trueforge-dev-bot[bot] <trueforge-dev-bot[bot]@users.noreply.github.com>
![qodex-ai[bot]](https://avatars.githubusercontent.com/u/8568200?s=60&v=4){kind=small}
Qodex reviewReviewed units u1. Documents closed-only ingestion and removal of pause_timeout_ms. Effort: ●●●●● (5/5) · 3 findings: 1 major, 2 minor Since last review: 0 addressed, 0 new. Scope: the changes since the last review (delta: only what changed since the previously reviewed commit 2db18f8 is reviewed; findings are anchored on the whole change). Qodex review threads: 3 open, 15 resolved by Qodex, 4 resolved by a reviewer. 3 findings still present from earlier reviews; inline comments not duplicated. See the collapsed detail. WalkthroughNotes
Still present from earlier reviewsThese findings were already reported on this PR and remain unresolved. The original inline comments are still attached at their first-seen lines.
What Qodex checkedLenses (openqodex review with checked answers, all dimensions)
Scanners
Quality filters
Verification (M1b probes against PR preview)
Cross-review dedup
Output
Reviewer: shown 55 of 55 changed hunks, opened 6 files (7 changed files never opened), 13 tool calls served, 6 model calls, 0 correction rounds, 55.5s. The library does not report tool calls asked apart from those it served, or a step cap. Every dimension is reviewed. Bugs, security, and performance post inline; low-severity style and convention notes are collected in the group above rather than dropped. Mark a finding resolved or flag a false positive in the Qodex Pull Requests view. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
Only leftover {sequence}-1 streams accept the legacy cursor; new streams
return 400 for after_sequence_number and Last-Event-ID. Drop the
skip-count compat path and mark subscribe non-resumable for Fern so the
SDK stops reconnecting with Last-Event-ID.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Partial repair cannot match emit order, so an existing stream is left untouched. Seeks compare a dedicated id field instead of decoding payloads, and the Redis owner key is removed so ownership stays on the turn row. Co-authored-by: Cursor <cursoragent@cursor.com>
…ward. A restore writes the durable log to a private key and renames it onto the stream only while the stream is still absent, so a stream is never left as a prefix. An after_event_id seek walks down from the tip, so a recent cursor settles in one step. Co-authored-by: Cursor <cursoragent@cursor.com>
Restore logs the missing-stream rebuild, each durable page query, the install or lost race, and failures. Counts and timings stay in the message text so they are not indexed as log fields. Request-scoped ids come from the log context. Co-authored-by: Cursor <cursoragent@cursor.com>
6527f22 to
e6556d2
Compare
| pages: AsyncIterable<readonly T[]>, | ||
| streamTTLSeconds: number | undefined, | ||
| ): Promise<boolean> { | ||
| const tempKey = `${this.streamId}:restore:${randomUUID()}`; |
There was a problem hiding this comment.
Medium: Bound concurrent log restoration
When a paused turn's stream has expired, parallel authorized /subscribe requests can all pass the absent-stream check and build complete copies under separate UUID keys; renameNX only selects a winner after those copies have been allocated, allowing a burst to multiply a large log's Redis footprint and pressure shared resources. Acquire a per-stream restore lease shared across replicas, recheck existence after acquiring it, and bound overall restore concurrency.
| } | ||
| await Promise.all(writes); | ||
| wrote = true; | ||
| await this.expireKey(tempKey, streamTTLSeconds); |
There was a problem hiding this comment.
🟡 Minor Partial restore after temporary key expiration · bug
installIfAbsent expires tempKey while durable page queries remain pending. If a query exceeds streamTTLSeconds, subsequent xAdd calls recreate tempKey without earlier pages. renameNX then installs the incomplete stream and reports success. Subscribers miss earlier restored events despite the successful restore result. Keep tempKey alive throughout page retrieval, or detect expiration and restart the complete restore before renameNX.
Bring in the main sync and pause migration ordering from ha/emit-pause. Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # packages/trueforge/src/apis/turns.ts
e6556d2 to
48a41df
Compare
| const reply: unknown = await this.redis.eval(PUT_STREAM_ENTRY, { | ||
| keys: [this.streamId], | ||
| arguments: [ | ||
| this.entryMillisecondFor(event), |
There was a problem hiding this comment.
🟠 Major Legacy resume cursors stop advancing · bug
put appends timestamp entries to existing legacy streams. readEvent returns these entries without sequence_number, although subscribe still accepts after_sequence_number. Legacy clients cannot advance their sequence cursors and replay the appended events on every reconnect. Preserve sequence-based writes and sequence_number for existing legacy streams until those streams expire.
| } | ||
| await Promise.all(writes); | ||
| wrote = true; | ||
| await this.expireKey(tempKey, streamTTLSeconds); |
There was a problem hiding this comment.
🟡 Minor Temporary restore expiry discards earlier pages · bug
installIfAbsent expires tempKey after each page. A subsequent page read exceeding streamTTLSeconds lets Redis delete earlier pages before xAdd recreates tempKey. renameNX installs incomplete history, and applyEventLogRestore preserves that incomplete stream on subsequent restores. Keep tempKey alive throughout page reads, or detect expiry and restart the entire restore before renameNX.
A resume starts at the oldest open message shell, and reconnect progress is counted per cursor so replaying the same events still exhausts the retry budget. Test fixtures include the memory catalog and configuration switch. Co-authored-by: Cursor <cursoragent@cursor.com>
…oss pauses - Pick the earliest open model message only from the current turn so a resume cursor from another turn cannot leak in. - Keep a finished model.message when a replayed shell arrives, so replays do not truncate tool arguments while approvals are pending. - Walk history newest-first for the reconnect cursor and pin it to mcp.auth_required until the auth prompt is answered or the turn moves on. Co-authored-by: Cursor <cursoragent@cursor.com>
…and early sandbox id - Restore pending_mcp_auth onto rebuilt AgentThreads so a turn paused for MCP auth does not run before user.mcp_auth_continue arrives. - Memory ingest treats a tip turn as paused when status is 'paused' and uses turn.updated_at as the pause time, matching the new paused state. - Carry the open tip's sandbox id onto the pre-getTurn progress paint. - Accept any state shape in cursorFromTurnEvents and narrow it locally. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2db18f8. Configure here.
A turn paused on the user can resume on the same turn id, so ingesting it early moved the checkpoint onto a turn whose final reply was then skipped. Only closed exchanges are ingested now; the pause timeout and partial sends are removed. Co-authored-by: Cursor <cursoragent@cursor.com>
Note
Medium Risk
Touches live turn streaming, subscribe API semantics, Redis stream layout, and memory ingestion selection—all user-visible if wrong. Changes are well-tested but span core runtime and persistence.
Overview
Turn subscribe and resume now use event-id cursors (
from_event_id/after_event_id), deprecating integerafter_sequence_numberon new streams; the API, OpenAPI, generated SDK, and UI adapter are updated accordingly, and SSE no longer advertises built-in reconnection.Server-side event streams were reworked: live Redis/in-memory logs use millisecond entry ids, can restore from the durable log when a paused turn’s stream is gone, validate resume cursors, and only attach legacy sequence numbers for old streams; create/subscribe paths and drain logic were updated, plus per-turn lock acquire timeout (
TURN_LOCK_ACQUIRE_TIMEOUT_MS).The assistant-ui runtime tracks subscribe cursors (including MCP auth and overlapping model messages), stores them on active streams, and passes them through
subscribeToTurn; sequence numbers are optional, and tests cover the new replay/progress behavior.Memory ingest now sends only closed exchanges, drops
pause_timeout_msand paused-range session scanning, and adjusts store queries and contracts; configuration/docs reflect the change.Agent threads carry
pendingMCPAuthwhen rehydrated so a rebuilt paused turn waits for user MCP continue without re-prompting; related session-store contract tests were extended.Reviewed by Cursor Bugbot for commit 455dfb1. Bugbot is set up for automated code reviews on this repo. Configure here.