Possible merging of thinking blocks from retry attempts

[Slava-256]

What's happening?

Sometimes I see Anthropic models taking too long to think, and then I see briefly a brown colored message from Big-AGI saying the it is retrying. The message disappears quickly, but I think it says something like [2/4 attempts].

If I click "show reasoning" to open up the thinking block after this happens, I see very long thinking text, and from the text it does look like the model started thinking through the question from the very beginning, producing similar output as there is at the very beginning of that long thinking block.

I wonder if Big-AGI failed to clear thinking from previous request attempt, or is there some Anthropic API feature that allows retry and then resume request that previously failed?

This is in beam mode, if it makes a difference.

Where does this happen?

Big-AGI Pro (big-agi.com)

Impact on your workflow

Low - Minor inconvenience

Environment (if applicable)

No response

Additional context

No response

[github-actions]

Thanks for the detailed report - this is a real observation backed by a known code-level concern.

Assessment: Confirmed - Thinking blocks accumulate across retries

When a retry occurs (e.g., Anthropic overloaded_error mid-stream), the thinking content from the failed attempt is preserved and the retry's thinking content gets appended on top of it. This explains the duplicated reasoning you're seeing.

Root Cause

All three retry paths use rShallClear: false, meaning accumulated fragments (including thinking blocks) are never cleared between attempts:

1. Server operation retry (src/modules/aix/server/dispatch/chatGenerate/chatGenerate.operation-retry.ts:100):

   rShallClear: false, // preserve particles from prior continuation turns
   

2. Server dispatch retry (src/modules/aix/server/dispatch/chatGenerate/chatGenerate.executor.ts:113):

   rShallClear: false
   

3. Client-level retry (src/modules/aix/client/ContentReassembler.ts:142):

   rShallClear: false, // TODO: check if this is correct; we shall clear, but at the same time we haven't tried to see
   

   There's literally a TODO in the code questioning this decision.

What happens

The ContentReassembler.onRetryReset() method (ContentReassembler.ts:743) only clears fragments when rShallClear is true - which currently never happens. So when Anthropic's API returns an overloaded_error mid-stream after the model has already been thinking, the thinking fragment from the failed attempt stays in accumulator.fragments, and the retry produces a new thinking fragment that gets appended alongside it.

Fix direction

The fix likely involves setting rShallClear: true for retry scenarios where the request starts from scratch (reconnect strategy), while preserving the current false behavior only for true resume scenarios where the server continues from where it left off. For Anthropic overloaded_error retries specifically, these are full restarts - so fragments should be cleared.

A nuanced approach:

• 'reconnect' retries (server overload, 429/503/502, Anthropic overloaded_error): rShallClear: true - the entire request restarts
• 'resume' retries (network interruption with a valid upstream handle): rShallClear: false - content continues from the interruption point

The brown message

The "[2/4 attempts]" placeholder you see briefly is the retry status indicator added at ContentReassembler.ts:759-766 - this is working as designed to inform the user.

[Slava-256]

I now see the issue extending even to the regular non-thinking output.

1st response got cut off in the middle, and then the new response started to append in the same window without clearing previous content.

[enricoros]

Initial analysis - the "Anthropic continuation" system, which requires to keep all content up to that point, and the "Reconnect/Retry" resilience system, which requires to dispose all content "up to" the last Anthropic continuation Breakpoint get desynchronized between server and client.

@claude Please understand both mechanisms deeply, and the 4 levels of reconnection resilience (client, to server, to trpc fetchers), and how they stack. Then verify the flow of particles. Then figure out the correct server-side nesting structure for the highest level of resilience and continuation and the client-side too. Then figure out mechanisms (breakpoints, bookmarks, etc) for synchronizing server-side and client side message accumulation on a per-AIX-call boundary.

[claude]

Claude finished @enricoros's task in 4m 54s —— View job

---

Deep Analysis: AIX Retry/Resilience & Anthropic Continuation Synchronization

• Understand the Anthropic continuation system (server-side)
• Understand the Reconnect/Retry resilience system (all 4 levels)
• Map the particle flow end-to-end (server -> client)
• Analyze how the 4 resilience levels stack/nest
• Identify the desynchronization between server and client accumulation
• Propose correct nesting structure for resilience + continuation
• Propose synchronization mechanisms (breakpoints/bookmarks)

---

1. The Two Mechanisms

Anthropic Continuation (pause_turn)

The continuation system preserves all accumulated content and tells the API to continue from where it stopped. It works at the semantic level - the model intentionally paused (not an error).

Flow (chatGenerate.continuation.ts):

1. Anthropic parser detects stop_reason === 'pause_turn' at anthropic.parser.ts:452
2. Throws DispatchContinuationSignal with a mutateBody() closure
3. Continuation wrapper catches it, chains a new dispatch creator that appends accumulated content blocks as an assistant message
4. Re-enters the operation retry loop with the mutated body
5. All previously-yielded particles are preserved - the client sees seamless continuation
6. Up to 10 continuation turns max

Key property: Content from ALL prior turns must be preserved on both server and client. The server sends the content back to Anthropic in the mutated body; the client's accumulated fragments represent that same content visually.

Retry/Resilience (error recovery)

Retries happen when something goes wrong - the entire request restarts from scratch. Content from the failed attempt should ideally be discarded because the retry will regenerate it (differently).

---

2. The Four Levels of Reconnection Resilience

Level	Scope	File	rScope	Trigger	What it retries
L1	HTTP Connection	trpc.fetchers.retrier.ts	srv-dispatch	DNS/TCP/429/502/503 before stream starts	The HTTP fetch itself
L2	Server Operation	chatGenerate.operation-retry.ts	srv-op	OperationRetrySignal from parsers (Anthropic overloaded_error, rate_limit_error, api_error mid-stream)	Entire dispatch (new HTTP request)
L3	Server Continuation	chatGenerate.continuation.ts	(info)	DispatchContinuationSignal (Anthropic pause_turn)	New dispatch with mutated body
L4	Client Low-Level	aix.client.ts + aix.client.retry.ts	cli-ll	tRPC/network errors (stream disconnect)	Entire tRPC call

How they nest (server-side call chain):

L4: Client retry loop (aix.client.ts:696 - while(true))
  └─ tRPC streaming call ──────────────────────────────────────┐
    L3: executeChatGenerateWithContinuation (continuation.ts:64)  │
      └─ L2: executeChatGenerateWithOperationRetry (op-retry.ts:55)
          └─ executeChatGenerateDispatch (executor.ts)              │
              └─ L1: fetchWithAbortableConnectionRetry (retrier.ts) │
                  └─ HTTP fetch to AI provider                      │

L4 is currently disabled (new AixStreamRetry(0, 0) at aix.client.ts:694).

How they stack:

• L1 (connection) retries the initial HTTP connection before any streaming starts. Once the response stream begins, L1 is done.
• L2 (operation) catches mid-stream errors thrown by parsers. It retries the entire dispatch (includes L1). Each L2 retry is a fresh HTTP connection to Anthropic.
• L3 (continuation) catches pause_turn. It creates a new dispatch with mutated body and re-enters L2. Each continuation turn gets its own fresh L2 retry budget (up to 4 attempts).
• L4 (client) would catch tRPC-level errors and retry the entire server chain (L3+L2+L1).

---

3. End-to-End Particle Flow

AI Provider (Anthropic SSE stream)
  │
  ▼
L1: fetchWithAbortableConnectionRetry ← retries HTTP fetch, emits srv-dispatch particles via callback
  │ Response object (stream not yet consumed)
  ▼
Executor: _consumeDispatchStream() loop
  │ read chunk → UTF-8 decode → SSE demux → parse events
  │ Parser calls ChatGenerateTransmitter methods (appendText, appendReasoningText, etc.)
  │ yield* chatGenerateTx.emitParticles()  ← particles flow up through generators
  │
  │ Parser may throw: OperationRetrySignal or DispatchContinuationSignal
  ▼
L2: executeChatGenerateWithOperationRetry
  │ yield* dispatch (transparent pass-through for particles)
  │ On OperationRetrySignal: yield srv-op retry particle → delay → loop
  ▼
L3: executeChatGenerateWithContinuation
  │ yield* L2 (transparent pass-through for particles)
  │ On DispatchContinuationSignal: yield flow-cont info particle → mutate body → loop
  ▼
tRPC streaming transport → network → client
  ▼
L4: aix.client.ts while(true) loop
  │ for await (particle of particleStream) → reassembler.enqueueWireParticle(particle)
  ▼
ContentReassembler
  │ #processWireBacklog() → #reassembleParticle() per particle
  │ Builds accumulator_LL.fragments[] incrementally
  ▼
onGenerateContentUpdate → UI renders fragments

---

4. The Desynchronization Problem

The root cause is that the server and client have two independent accumulators that must stay in sync, but the retry mechanism breaks this contract.

Server-side accumulator

The Anthropic parser maintains responseMessage.content - an array of content blocks accumulated during streaming. When pause_turn fires, this array is passed to _createAnthropicPauseTurnContinuation() which converts it into the mutateBody() closure. The server discards its per-dispatch state on each new dispatch - the accumulated content lives in the mutated request body.

Client-side accumulator

accumulator_LL.fragments[] in aix.client.ts:675. This persists across retry iterations because the accumulator object lives outside the while-loop, and a new ContentReassembler is created each iteration but given the same accumulator (aix.client.ts:717-718).

The desynchronization scenarios

Scenario A: L2 retry mid-stream (the reported bug)

Turn 0 (first dispatch):
  Server: streams thinking + text → client accumulates [thinking_frag, text_frag]
  Server: receives overloaded_error mid-stream
  Server: throws OperationRetrySignal
  Server: yields { cg: 'aix-retry-reset', rShallClear: false }  ← PROBLEM
  Client: sees rShallClear=false → keeps [thinking_frag, text_frag, retry_placeholder]

Turn 0 retry (second dispatch):
  Server: starts fresh HTTP request, parser state is fresh
  Server: streams thinking + text again (model starts over)
  Client: appends [thinking_frag_2, text_frag_2] to existing fragments

Result: Client has [thinking_frag, text_frag, retry_ph, thinking_frag_2, text_frag_2]
        → Duplicate thinking/text content!

Scenario B: L2 retry during a continuation turn (worse)

Turn 0: completes normally, pause_turn
  Client: [content_from_turn_0]
  Server: mutates body with turn_0 content

Turn 1 (continuation):
  Server: streams new content → client appends [content_from_turn_1_partial]
  Server: overloaded_error mid-stream
  Server: throws OperationRetrySignal, yields { rShallClear: false }
  Client: keeps [content_from_turn_0, content_from_turn_1_partial, retry_ph]

Turn 1 retry:
  Server: restarts dispatch with SAME mutated body (turn_0 content baked in)
  Server: streams turn_1 content again from scratch
  Client: appends [content_from_turn_1_retry]

Result: Client has [turn_0, turn_1_partial, retry_ph, turn_1_retry]
        → turn_1 content duplicated (partial + complete)
        But server body only has turn_0 → server and client are in sync
        for what the model sees, just the client display is wrong

Scenario C: L4 client retry (if enabled)

Server: completes multiple continuation turns, streams lots of content
tRPC connection drops
Client: has [all_accumulated_fragments]
Client: reconnects (new tRPC call)
Server: starts completely fresh (no continuation state, no body mutation)
Server: streams response from scratch
Client: appends NEW response to OLD fragments

Result: Complete duplication of all content

The fundamental tension

The rShallClear: false comment at operation-retry.ts:100 says:

"preserve particles from prior continuation turns; operation errors fire early with low/no particles"

This is the crux: the flag is false to protect continuation content, but it also protects the current attempt's partial content which should be cleared.

---

5. Correct Nesting Structure

The current nesting is actually correct architecturally. The issue isn't structural - it's that the retry signal doesn't carry enough information for the client to know what to clear.

Server-side: Correct as-is

L3 (continuation) wraps L2 (operation) wraps L1 (connection)

This is correct because:

• Each continuation turn should get its own retry budget (L2 resets per turn)
• Connection retries (L1) happen before streaming, so they don't affect content
• Operation retries (L2) restart the dispatch, which means the parser starts fresh

Client-side: Needs breakpoint awareness

The client needs to know: "when you get an L2 retry, clear everything since the last continuation boundary, but keep everything before it."

L4 client retry: Two strategies needed

When L4 is enabled:

• Resume (with upstream handle): The server continues from where it left off. Client should keep all content. rShallClear: false is correct.
• Reconnect (no handle): The server starts completely fresh. Client MUST clear all content. rShallClear: true is correct.

---

6. Proposed Synchronization Mechanism: Continuation Breakpoints

The core idea: introduce a breakpoint index that marks the boundary between "confirmed content from prior continuation turns" and "current attempt content that may need to be discarded."

6.1 New particle: aix-continuation-breakpoint

Emitted by L3 before each new continuation turn begins:

// In chatGenerate.continuation.ts, just before re-entering L2:
yield { cg: 'aix-continuation-checkpoint' };

Wire type addition:

| { cg: 'aix-continuation-checkpoint' }

6.2 Client-side breakpoint tracking

In ContentReassembler:

private continuationCheckpointIndex: number = 0; // fragments index at last checkpoint

When aix-continuation-checkpoint is received:

private onContinuationCheckpoint(): void {
  // Everything up to this point is confirmed content from completed turns
  this.continuationCheckpointIndex = this.accumulator.fragments.length;
}

6.3 Changed rShallClear semantics → rClearStrategy

Replace the boolean rShallClear with a discriminated strategy:

| { cg: 'aix-retry-reset',
    rScope: 'srv-dispatch' | 'srv-op' | 'cli-ll',
    rClearStrategy: 'none' | 'since-checkpoint' | 'all',
    // ... rest unchanged
  }

• 'none': Keep everything (L1 connection retries - no content streamed yet)
• 'since-checkpoint': Clear fragments from continuationCheckpointIndex onward (L2 operation retries)
• 'all': Clear everything (L4 reconnect)

6.4 Updated retry reset handler

private onAixRetryReset({ rScope, rClearStrategy, ... }): void {
  switch (rClearStrategy) {
    case 'none':
      break; // keep everything

    case 'since-checkpoint':
      // Clear content from current attempt, preserve prior continuation turns
      this.accumulator.fragments.length = this.continuationCheckpointIndex;
      this.currentTextFragmentIndex = null;
      this.wireParticlesBacklog.length = 0;
      break;

    case 'all':
      // Full wipe for reconnect scenarios
      this.accumulator.fragments = [];
      this.continuationCheckpointIndex = 0;
      this.currentTextFragmentIndex = null;
      delete this.accumulator.legacyGenTokenStopReason;
      this._terminationReason = undefined;
      this._tokenStopReasonWire = undefined;
      this.wireParticlesBacklog.length = 0;
      break;
  }

  // Show retry placeholder (always)
  // ...existing placeholder code...
}

6.5 What each level emits

Level	When	rClearStrategy	Why
L1 connection retry	Before stream starts	'none'	No content has been streamed yet
L2 operation retry	Mid-stream error	'since-checkpoint'	Discard partial current-turn content, keep prior turns
L3 continuation	After pause_turn	(emits checkpoint, not retry)	Marks boundary between turns
L4 reconnect	tRPC disconnect, no handle	'all'	Server starts from scratch
L4 resume	tRPC disconnect, has handle	'none'	Server continues from where it left off

6.6 Implementation changes summary

Server-side (3 files):

1. aix.wiretypes.ts: Add aix-continuation-checkpoint control op. Replace rShallClear: boolean with rClearStrategy: 'none' | 'since-checkpoint' | 'all'.

2. chatGenerate.continuation.ts:94: Before the info particle, yield { cg: 'aix-continuation-checkpoint' }.

3. chatGenerate.operation-retry.ts:100: Change rShallClear: false to rClearStrategy: 'since-checkpoint'.

4. chatGenerate.executor.ts:114: Change rShallClear: false to rClearStrategy: 'none' (connection retry happens before streaming).

Client-side (2 files):

5. ContentReassembler.ts: Add continuationCheckpointIndex field. Handle aix-continuation-checkpoint. Update onAixRetryReset to use rClearStrategy.

6. aix.client.ts:170 (if L4 enabled): Use rClearStrategy: rsm.resumeHandle ? 'none' : 'all'.

6.7 Edge cases

Checkpoint at turn 0: continuationCheckpointIndex starts at 0, so 'since-checkpoint' on the first turn clears everything - correct behavior (no prior turns to preserve).

Multiple L2 retries within one turn: Each retry emits 'since-checkpoint', which truncates back to the same checkpoint - correct, only the latest attempt's content is kept.

L2 retry across turns with no continuation: If the model never pauses, checkpoint stays at 0. A 'since-checkpoint' clears all content - correct, same as 'all' when there are no prior turns.

Void placeholders: The retry placeholder is appended after the truncation, so it appears at the end of confirmed content - correct.

Metrics/model/handle: These live outside fragments[] and should be preserved across all retry types (they're metadata about the connection, not content).

---

7. Alternative Considered: Fragment Tagging

Instead of a checkpoint index, each fragment could carry a continuationTurn: number tag. On retry, fragments with the current turn number get removed. This is more robust against fragment reordering but adds overhead to every fragment creation. The checkpoint approach is simpler and sufficient given that fragments are append-only.

---

8. Visual Summary of the Fix

Before (current behavior - broken):
  [turn0_thinking][turn0_text][retry_ph][turn0_thinking_dup][turn0_text_dup]
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                         Should have been cleared

After (with checkpoints):
  Checkpoint=0
  L2 retry → truncate to index 0 → [retry_ph]
  Retry streams → [retry_ph][turn0_thinking][turn0_text]

  pause_turn → checkpoint moves to 3
  L2 retry → truncate to index 3 → [retry_ph_removed][turn0_thinking][turn0_text][retry_ph]
  Retry streams → [...turn0...][retry_ph][turn1_thinking][turn1_text]

---

Summary

The issue is that the server-side and client-side accumulators diverge on retry because rShallClear: false was chosen to protect continuation content, but it also protects stale content from failed attempts. The fix is a continuation checkpoint mechanism:

1. Server emits a checkpoint particle at each continuation boundary
2. Client tracks the checkpoint as a fragment index
3. Operation retries (L2) clear "since checkpoint" instead of "nothing" or "everything"
4. Client retries (L4) clear "all" on reconnect, "nothing" on resume

This preserves the continuation semantics while correctly discarding stale retry content.

---

[enricoros]

@claude Go with the proposed changes to the protocol, server and client. On the client side, also have a good placeholder particle to signal the continuation breakpoint (shall have a flow namespace already) and make sure that the breakpoint indexing is respected when back-searching, meaning things may not be touched before the breakpoint index during generation, and maybe a pass to solidify across breakpoint boundaries (e.g. fusing text, re-pairing tool calls) shall be done only upon completion, so we really make sure we are "rewindable" at any point in time?

[claude]

Claude finished @enricoros's task in 7m 54s —— View job • claude/issue-1055-20260402-1631 • Create PR ➔

---

Continuation Breakpoint Mechanism - Implemented

• Read all relevant source files (protocol, server, client)
• Protocol: Add aix-continuation-checkpoint particle & replace rShallClear with rClearStrategy
• Server: Emit checkpoint in continuation wrapper
• Server: Update operation-retry to use rClearStrategy: 'since-checkpoint'
• Server: Update executor dispatch retry to use rClearStrategy: 'none'
• Client: Track breakpoint index in ContentReassembler
• Client: Handle checkpoint particle, update retry reset logic with 'none' | 'since-checkpoint' | 'all'
• Client: Respect breakpoint during back-searching (reasoning append, signature set, redacted data, tool call args, placeholder lookup)
• Client: Solidify across breakpoint boundaries on completion (fuse adjacent text/reasoning fragments, remove checkpoint markers)
• Client: Update setClientRetrying for L4 retry strategy (reconnect -> 'all', resume -> 'none')
• Type-check passes (no new errors)
• Commit and push

Summary of changes (6 files, +140/-29 lines)

Protocol (aix.wiretypes.ts):

• New aix-continuation-checkpoint control op emitted at continuation boundaries
• rShallClear: boolean replaced with rClearStrategy: 'none' | 'since-checkpoint' | 'all' on aix-retry-reset
• aix-info ait union extended with 'flow-checkpoint'

Server (3 files):

• chatGenerate.continuation.ts - Emits aix-continuation-checkpoint before each continuation turn, before the flow-cont info particle
• chatGenerate.operation-retry.ts - Uses rClearStrategy: 'since-checkpoint' to discard current-attempt content but preserve prior turns
• chatGenerate.executor.ts - Uses rClearStrategy: 'none' for connection retries (no content streamed yet)

Client (ContentReassembler.ts):

• continuationCheckpointIndex tracks the fragment boundary between confirmed and in-progress content
• onContinuationCheckpoint() sets the checkpoint, breaks text accumulation, and adds a visible flow-checkpoint placeholder
• onAixRetryReset() now uses 3-way strategy: none (keep all), since-checkpoint (truncate to checkpoint), all (full wipe)
• Back-search protection: onAppendReasoningText, onSetReasoningSignature, onAddRedactedDataParcel, onAppendFunctionCallInvocationArgs, onSetOperationState, and _removeLastVoidPlaceholderDelayed all respect the checkpoint boundary - never modifying frozen content from prior turns
• Solidification pass (_solidifyFragmentsAcrossBoundaries) runs at finalization: fuses adjacent text fragments, fuses adjacent reasoning fragments (merging signatures and redacted data), and removes checkpoint placeholder markers
• setClientRetrying uses 'all' for reconnect (server restarts from scratch) and 'none' for resume

Fragment types (chat.fragments.ts):

• ac-info ait extended with 'flow-checkpoint' for the checkpoint placeholder display

|

[enricoros]

Worked on this manually. Still verifying and NOT propagated to the production site yet.

[enricoros]

@Slava-256 This is now fixed and works well in my testing. Thanks for reporting it - I ended up doing a large manual rewrite to support reconnections even within the progressive Anthropic continuation (pause return code) system. 👍

[Slava-256]

@enricoros Thank you! It was a sneaky issue because it required close inspection of the output to notice that there is something wrong with it.

[enricoros]

@Slava-256 Thank you again. One note: I am redoing the core of the AI communication protocol to enable top-class tools support and agentic behavior. Step #1 was to add reliability to the whole stack, and I worked on it on the weekend. Please let me know if you see any change of behavior in the coming week and I'll look into it promptly.

[enricoros]

@Slava-256 New: files created in contained by Anthropic are now downloadable. So whenever Claude decides to write a file with a report instead of showing it as text -- which happens more than I'd like to see Claude doing -- at least now you can download the file.

In addition, the Containers where code/skills run, are persisted by Anthropic for up to 1 hour and reused by Big-AGI in the current conversation. So if you execute commands such as "ls /tmp and tell me what's there" it will work across chat messages.