Vision Precision Bench (VPB)

Fine-grained visual accuracy — not caption fluency

Version 0.2.0 · 511 frozen synthetic cases · 6 tracks · transparent VPI

Public site (Keel) · Leaderboard · Versioning · Research · Design · Changelog · Pilot

---

The problem

Frontier VLMs sound fluent. That is not the same as seeing precisely.

Can the model:

• notice a tiny visual change and point to it?
• read 6–12px text in Chinese and English?
• tell O from 0, 日 from 曰, emoji variants?
• count objects under clutter?
• name colors and compare sizes?

VPB measures these constructs separately, then composes a disclosed index.

Inspired by public board craft from SWE-bench (Lite / Verified / versioned splits) and DeepSWE (contamination-aware tasks, reliable verification, leaderboard-first presentation).

---

What’s in v0.2.0

Track	Cases	Primary metric	Difficulty axes
change_spot	128	F1@0.5 (+ loose/center hit)	easy → micro
text_read	126	0.7·EM + 0.3·char-acc	size 24→6px · lang · contrast
text_confusable	128	exact	O/0 · CJK lookalikes · emoji
count	48	exact integer	n=3…20
color	33	exact name	pure / noisy / small
relative_size	48	left/right	ratio 1.8→1.05
Total	511	VPI	multi-track

Freeze SSOT: data/VPB_VERSION.json

Composite index (VPI)

Equal-weight mean of track primaries by default.
Profiles: balanced · change_heavy · ocr_heavy.

Rules: always publish per-track scores · missing track ⇒ VPI is null · report n.

---

Pilot (harness sanity) — Grok 4.5

Track	Score
change_spot F1@0.5	0.00
text_read	0.87
text_confusable	0.63
count	1.00
color	1.00
relative_size	0.92
VPI (equal)	0.74

Read: easy color/count/large-text PASS → harness works.
Tiny text + confusable drop → discrimination works.
Strict change F1@0.5 alone is harsh when models see but box poorly — dual localization metrics are first-class.

Details: results/pilot/grok-4.5/PILOT_INTERPRETATION.md

---

Quick start

git clone https://github.com/SylphxAI/spot-diff-bench.git
cd spot-diff-bench
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt openai Pillow
export PYTHONPATH=.

# change_spot baselines (no API key)
.venv/bin/python -m spot_diff_bench run \
  --dataset data/synthetic \
  --adapter pixel_oracle \
  --out results/oracle

# multi-track pilot via OpenRouter
export OPENROUTER_API_KEY=...
export OPENAI_API_KEY=$OPENROUTER_API_KEY
export OPENAI_BASE_URL=https://openrouter.ai/api/v1
.venv/bin/python scripts/pilot_frontier.py \
  --model x-ai/grok-4.5 \
  --out results/pilot/my-run

Regenerate the frozen split (must match seeds for v0.2.0):

.venv/bin/python -m spot_diff_bench generate --out data/synthetic --n 128 --seed 42
.venv/bin/python - <<'PY'
from pathlib import Path
from spot_diff_bench.tracks.suite import generate_all_tracks
# note: count/size sizes in suite helpers; prefer committed data/ for exact freeze
print('use committed data/ for leaderboard claims')
PY

---

Versioning (required for any public claim)

Release	Role	Cases
0.1.0	SpotDiff scaffold	48
0.2.0	Public multi-track preview	511
0.3.x	Standard denser cells	~1k–1.5k (planned)
1.0.0	Citation bar + real annotated pack	≥2k (planned)

Full policy: docs/VERSIONING.md · history: CHANGELOG.md

If you omit the version, your number is not a VPB result.

---

Why ~500 now (not 50, not 50k)

Too few	Too many
unstable ranks	unaffordable multi-model matrices
empty difficulty cells	slow iteration / silent saturation
marketing-only	hard to audit

~500 multi-track matches the “Lite/Verified” public-board pattern: enough cells for curves, cheap enough to run many models, honest as preview before 1.0.

---

Sample fixtures

change_spot · easy (GT in red)

change_spot · micro

text_read · Chinese sample

---

Repository layout

VERSION                 # 0.2.0
CHANGELOG.md
CITATION.cff
data/VPB_VERSION.json   # freeze SSOT
data/synthetic/         # change_spot
data/vpb/               # other tracks
leaderboard/            # board + baselines
spot_diff_bench/        # harness
scripts/pilot_frontier.py
docs/                   # research, design, versioning
results/pilot/          # public pilot evidence

---

Contributing results

1. Run v0.2.0 freeze
2. Open PR: leaderboard/runs/<slug>/score.json
3. Include model id, date, adapter, temperature, tools

See CONTRIBUTING.md · leaderboard/LEADERBOARD.md

---

Citation

@software{vpb_0_2_0,
  title  = {Vision Precision Bench (VPB) v0.2.0},
  author = {SylphxAI},
  year   = {2026},
  url    = {https://github.com/SylphxAI/spot-diff-bench},
  note   = {511 synthetic cases across 6 vision accuracy tracks}
}

---

License

MIT © SylphxAI

Related: Iris / image-reader-mcp — local evidence-first image facts for agents.