Skip to content

fix: clean pause on transient Ollama capacity/overload without burning escalation#86

Merged
tzone85 merged 1 commit into
mainfrom
fix/ollama-capacity-clean-pause
Jun 24, 2026
Merged

fix: clean pause on transient Ollama capacity/overload without burning escalation#86
tzone85 merged 1 commit into
mainfrom
fix/ollama-capacity-clean-pause

Conversation

@tzone85

@tzone85 tzone85 commented Jun 24, 2026

Copy link
Copy Markdown
Owner

Problem

Running multiple concurrent builds against one Ollama instance, the server returns transient overload/resource conditions: HTTP 429/503, `server busy`, `no slots available`, `maximum pending requests exceeded`, `model is loading`, `out of memory`, `context deadline exceeded`, `connection refused`. These were never classified, so the pipeline treated them as story-quality failures and only paused the requirement after burning the whole escalation chain (reset → manager diagnosis → tech-lead re-plan), with misleading messages ("agent produced no code changes", "re-plan failed").

Fix

  • `internal/llm/errors.go` — `IsCapacityError(err)` classifies 429/503/529 (typed `*APIError`) and stringified Ollama HTTP-client errors carrying a capacity signature. `ContainsCapacitySignature(string)` is the shared vocabulary. Capacity is distinct from `IsFatalAPIError` (401/403/billing — permanent) and from a 404 model-not-found (operator config error). Signatures are the exact strings `internal/llm/ollama.go` and the Ollama server emit.
  • `internal/engine/capacity_pause.go` — `Monitor.pauseIfCapacity(storyID, stage, err)` pauses the requirement without consuming an escalation attempt or advancing the tier. Wired into: reviewer, merge/conflict-resolution, manager diagnosis, tech-lead re-plan. `agentCompletionHasCapacityError(storyID)` scans the latest `STORY_COMPLETED` payload's `error` field so a native (Gemma) agent that hit an overload (empty diff) pauses cleanly instead of escalating as "no code changes". The pause reason states the cause is transient — resume after the server recovers.

Tests

  • `internal/llm/capacity_test.go` — classifier coverage (signatures + non-signatures, not-fatal invariant).
  • `internal/llm/ollama_test.go` — a live 503/429 from the Ollama server is classified as capacity, not fatal.
  • `internal/engine/capacity_pause_test.go` — `pauseIfCapacity` (pauses without escalation / returns false otherwise), `agentCompletionHasCapacityError`, and post-execution review + empty-diff paths assert REQ_PAUSED with zero STORY_REVIEW_FAILED.

TDD throughout: tests written first, watched fail, then implemented.

Green gate

```
go build -o /tmp/nxd ./cmd/nxd && go vet ./... && go test ./... -count=1 # exit 0, all 30 packages ok
golangci-lint run ./... # 0 issues
```

…g escalation

Transient Ollama conditions (HTTP 429/503/529, "server busy", "no slots
available", "model is loading", "out of memory", "context deadline
exceeded", "connection refused") were treated as story-quality failures,
so a requirement only paused after burning reset -> manager -> tech-lead
re-plan with misleading messages.

- internal/llm/errors.go: IsCapacityError + ContainsCapacitySignature
  classify 429/503/529 typed errors and the stringified Ollama HTTP-client
  envelopes. Distinct from IsFatalAPIError (permanent) and 404 (config).
- internal/engine/capacity_pause.go: Monitor.pauseIfCapacity pauses the
  requirement without consuming an escalation attempt; wired into reviewer,
  merge/conflict-resolution, manager diagnosis, tech-lead re-plan, and the
  native empty-diff path via agentCompletionHasCapacityError (scans the
  STORY_COMPLETED payload error field).

Tests: internal/llm/capacity_test.go, ollama_test.go overload case,
internal/engine/capacity_pause_test.go (unit + post-execution paths).
@tzone85 tzone85 merged commit dd2c8c2 into main Jun 24, 2026
10 checks passed
@tzone85 tzone85 deleted the fix/ollama-capacity-clean-pause branch June 24, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant