judge: clamp max_tokens to the model's output cap (gpt-4o-mini 400s on 32768)

[saikrishnarallabandi]

Problem

LLMJudge hard-codes max_tokens=32768 at two call sites:

• llm_judge.py#L142 — _call(...)
• llm_judge.py#L337 — evaluate(...)

gpt-4o-mini — the default in config.example.yaml — accepts at most 16,384 completion tokens. OpenAI rejects the request:

400 max_tokens is too large: 32768. This model supports at most 16384 completion tokens,
whereas you provided 32768.

Because the 400 is inside the max_retries=20 loop, the judge retries it 20 times with ~8s backoff (~160s per judge call). Grading then exceeds the batch's patience and the trial is dropped with no error surfaced — the trace ends up with no grading_result at all.

Why it isn't caught with a local judge

An ollama-backed judge silently ignores an oversized max_tokens, so this only appears when the judge is an actual OpenAI endpoint. We hit it the moment we pointed judge.base_url at api.openai.com to match the published evaluation setup.

It presents exactly like an invalid API key — no grading output, no traceback. It cost us a while to find, since the key was fine.

Reproduce

# config.yaml: judge.model_id: gpt-4o-mini, judge.base_url: https://api.openai.com/v1
claw-anything batch --tasks-dir benchmark/tool --filter T04_T05 \
  --model <any> --agent loop --parallel 1

The trace file ends with trace_end and no grading_result. Run claw-anything grade on it in the foreground and the cause is visible:

[judge-retry] (400), attempt 7/20, waiting 8.1s ...

Minimal confirmation of the underlying limit:

curl https://api.openai.com/v1/chat/completions -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H 'content-type: application/json' \
  -d '{"model":"gpt-4o-mini","max_tokens":32768,"messages":[{"role":"user","content":"ok"}]}'

Fix

Clamp both call sites to min(max_tokens, 16384). No behaviour change for backends that accept larger values, since judge responses are far below either cap.

Environment

OS	Ubuntu 22.04.5 LTS, kernel 6.8.0-124
Python	3.12.10
Docker	29.1.3 (trials run --trial-in-container)
openai SDK	2.47.0
Judge	gpt-4o-mini via https://api.openai.com/v1
Agent model	Qwen3.5 (0.8B / 4B) via ollama 0.31.2
Base commit	verified on f9a0f63 and current main (e19dce2)

After the fix: 0 retries, grading completes, and a task that previously produced no grading_result scores normally.