Skip to content

fix: handle serde duplicate field error when providers send both legacy and new usage field names#331

Merged
ajianaz merged 2 commits into
developfrom
fix/serde-usage-duplicate-field
Jul 2, 2026
Merged

fix: handle serde duplicate field error when providers send both legacy and new usage field names#331
ajianaz merged 2 commits into
developfrom
fix/serde-usage-duplicate-field

Conversation

@ajianaz

@ajianaz ajianaz commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Problem

GPT-5.4 returns both legacy (prompt_tokens) and new (input_tokens) field names in the usage object simultaneously. serde_json >= 1.0.120 tracks visited fields during struct deserialization — when an alias maps to a field already set from the primary key, it raises duplicate field error, breaking all reviews using affected models.

Closes #330

Changes

Before: Usage struct used #[serde(alias = "input_tokens")] to accept both naming conventions. When both names appeared in the same JSON, serde errored.

After: Usage is now a plain struct (no serde derive). New parse_usage_value() function extracts from serde_json::Value with explicit preference order:

  1. Primary: prompt_tokenscompletion_tokenstotal_tokens
  2. CamelCase: promptTokenscompletionTokenstotalTokens
  3. New (GPT-5+): input_tokensoutput_tokens

This handles all three scenarios:

  • Legacy only (GPT-4, Claude) — works as before
  • New only (some future providers) — resolved correctly
  • Both (GPT-5.4) — deduplicated, no error

Both streaming (extract_stream_usage) and non-streaming (ChatResponse) code paths updated.

Tests

  • 6 new tests covering all usage field naming scenarios
  • All 609 existing tests pass (581 unit + 16 CLI + 6 config + 6 new)

Cora Self-Review Notes

Cora flagged 2 theoretical concerns on this PR:

  1. Zero-valued legacy fields — would only matter if a provider sent prompt_tokens: 0 with input_tokens > 0. No known provider does this; GPT-5.4 sends matching values.
  2. u64→u32 truncation — requires >4.2B tokens in a single request, physically impossible.

Both are acceptable trade-offs for now. Can revisit if a provider actually exhibits these behaviors.

…cy and new usage field names

GPT-5.4 (and potentially other newer models) return both legacy
(prompt_tokens/completion_tokens) and new (input_tokens/output_tokens)
field names in the usage object simultaneously.

serde_json >= 1.0.120 tracks visited fields during struct
deserialization. When an alias maps to a field already set from the
primary key, it raises 'duplicate field' error, breaking all reviews
using affected models.

Fix: Replace serde alias-based Usage deserialization with manual
parse_usage_value() that extracts from serde_json::Value with explicit
preference order (primary > camelCase > new). Updated both streaming
and non-streaming code paths.

Closes #330
@ajianaz ajianaz added bug Something isn't working priority:high scope:core Core review engine labels Jul 2, 2026
@ajianaz ajianaz merged commit ceaf22b into develop Jul 2, 2026
10 checks passed
@ajianaz ajianaz deleted the fix/serde-usage-duplicate-field branch July 2, 2026 02:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority:high scope:core Core review engine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: serde_json duplicate field error on providers that send both legacy and new usage field names (GPT-5.4)

1 participant