Entity Coverage Check

An adversarial multi-agent verifier that reconciles an authoritative reference list against a scraped dataset to find the entities you forgot to capture — built for fuzzy, transliterated Arabic project names where a shared district word is a trap, not a match.

You have two lists of the same real-world things, written by different people in different systems: an authoritative reference registry (the source of truth) and a captured pipeline (what your scraper actually pulled from the market). Naive string similarity fails here — the hard cases score higher than the easy ones. This tool pairs deterministic rapidfuzz pre-scoring with a defensively designed LLM pipeline (two independent judges, an auditor, and adversarial finders with a tie-breaker) and emits a styled per-region Excel report of every genuine coverage gap.

The worked example throughout is Saudi residential real estate, where project names are English transliterations of Arabic and vary wildly in spelling.

---

Why it's interesting

Most "does list A cover list B?" scripts stop at a fuzzy-match threshold. That is exactly wrong for transliterated entity resolution, and this repo makes the failure visible: run the naive scorer on the bundled sample and the three hardest false positives all score 92–100, while two genuine matches sit in the ambiguous band.

Registry - Northgate
  PRESENT  [100] Nawar                        -> Nawar(100) ; Adwar Villa(89)
  AMBIG    [ 76] Saraya Homes                 -> Saraya Al Jewan(76) ; Saraya Al Ruba 2(74)
  PRESENT  [100] Awali Al Fursan              -> Al Fursan Villas(100)      <- FALSE: shared district word
  PRESENT  [ 97] Durrat Al Madina 1           -> Durrat Al Madina 2(97)     <- FALSE: different phase
Registry - Southbank
  PRESENT  [ 91] Jadya Residences             -> Jadaya Residence(91)       <- true: transliteration variant
  PRESENT  [ 92] Assalah Community            -> Taj Al Asalah(92)          <- FALSE: shared district word

• The score is a hint, never a verdict. A 100% match driven by a shared district word (Awali Al Fursan vs Al Fursan Villas) is a false signal. The whole design exists to distinguish the distinctive brand token from the district/generic noise around it.
• Maker / checker separation, applied to LLM judgement. Two independently-prompted judges classify every project; an auditor re-attacks their agreements for false positives; adversarial finders re-attack the doubts. No single model call decides anything important alone.
• Adversarial by construction. A project is only ruled MISSING after finders were explicitly told to "try HARD to find it before concluding it is missing" — the cheapest way to keep recall honest is to make the finders want to prove the pessimist wrong.
• Ships a real deliverable. Not a console dump — a styled multi-tab workbook (summary + one tab per region) with the nearest pipeline name and the reason each gap was flagged, ready to hand to a non-technical stakeholder.

How the verdict is reached, precisely

The pipeline is deterministic up to the LLM stage; the LLM stage is where the judgement lives. Every rule below is in the code, not hand-waving.

Pre-scoring (common.py). Each reference name is scored against every pipeline name as the max of four rapidfuzz metrics — token_sort_ratio, token_set_ratio, partial_ratio, WRatio — after a normalisation pass that folds accents (NFKD), lowercases, strips punctuation, and drops a stop-word list of generic terms (villas, tower, residence, community, phase, al, …). Taking the most optimistic scorer is deliberate: it maximises recall of candidates, which the LLM then filters for precision. The top-4 candidates per project are passed downstream as hints.

The multi-agent verdict (coverage.wf.js), per region:

Stage	Who	Rule
Judge	2 independent judges (prompt variants A & B)	Each classifies every reference project PRESENT / MISSING with a confidence and a reason. Variant A walks the reference list; variant B isolates the distinctive token first and scans the pipeline for it.
Reconcile	deterministic	PRESENT only if both judges say PRESENT and min(confidence) ≥ 0.7 → consensus-present. Everything else → doubtful.
Audit	1 auditor	Re-examines the consensus-present pairs and flags any that align only on a shared district / suburb / generic word. Flagged pairs are demoted back into the doubtful set.
Find	2 adversarial finders (+1 tie-breaker)	Each doubtful project gets 2 independent finders told to try hard to locate it. Agree → done. Disagree → a 3rd adjudicates; final verdict is majority of the votes cast (present iff ≥ 2 of up to 3 say so).
Record	writer agent	The per-region verdict is persisted verbatim to wf/result_<slug>.json.

Every LLM call is constrained by a JSON Schema (JUDGE_SCHEMA, AUDIT_SCHEMA, FIND_SCHEMA) so the orchestrator gets typed objects back, not prose to parse. The same RULES block (transliteration equivalences, the district-word blacklist, the "phase numbers matter" clause) is injected into every prompt so all four agent roles reason under identical constraints.

How it works

reference.xlsx              *_pipeline.xlsx
(authoritative registry)    (scraped capture, one file/region)
        │                          │
        └──────────┬───────────────┘
                   ▼
        extract.py  ──►  data.json            {reference:{sheet:[…]}, pipeline:{file:[…]}}
                   │
                   ▼
        fuzzy.py    ──►  fuzzy.json           top-4 rapidfuzz candidates per project (hints)
                   │
                   ▼
        build_args.py ─► wf_args.json         reference metadata + hints, paired per region
                   │
                   ▼
        split_regions.py ─► wf/region_<slug>.json  +  wf/manifest.json
                   │
                   ▼
   coverage.wf.js   (agent-orchestration runtime, per region)
     Judge ─► Reconcile ─► Audit ─► Find ─► Record
                   │
                   ▼
        build_deliverable.py ─► coverage_gaps_by_region.xlsx   (summary + one tab per region)

• common.py — normalisation, the max-of-four fuzzy scorer, and pair_regions() (matches each reference sheet to its pipeline file on the shared region token, so nothing is hard-coded to one dataset).
• extract.py / fuzzy.py / build_args.py / split_regions.py — the deterministic Python pre-processing chain (Excel → JSON → hints → per-region workflow inputs).
• coverage.wf.js — the flagship: the adversarial multi-agent workflow described above.
• build_deliverable.py — renders the styled openpyxl workbook from the per-region results.

Runtime contract

coverage.wf.js targets a small, host-provided agent-orchestration runtime rather than any specific SDK. It expects five globals — swap in your own LLM provider behind agent():

Global	Contract
agent(prompt, {schema, phase, label})	Call an LLM; return an object validated against schema (a JSON Schema). With no schema, returns free text.
parallel(fns)	Run an array of async thunks concurrently; resolve to the array of results.
pipeline(items, ...stages)	Map items through sequential async stages, threading each stage's output into the next.
phase(name) / log(msg)	Progress + logging hooks.

The agent() runtime holds its own LLM credentials; no API keys live in this repo.

Tech stack

Python · rapidfuzz · openpyxl · JavaScript (agent-orchestration workflow) · JSON-Schema-constrained LLM calls

Running it locally

No real data ships with this repo (see Data & privacy). A tiny synthetic sample under sample/ lets you run the whole deterministic chain end-to-end; the LLM stage needs the agent runtime described above.

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# --- deterministic chain on the bundled synthetic sample ---
cp sample/data.json data.json     # or run extract.py against your own Excel inputs
python3 fuzzy.py                   # -> fuzzy.json  (and the naive-score preview above)
python3 build_args.py             # -> wf_args.json
python3 split_regions.py          # -> wf/region_*.json + wf/manifest.json

# --- adversarial verification (needs the agent-orchestration runtime) ---
# run coverage.wf.js in your runtime with wf/manifest.json as its args
# -> writes wf/result_<slug>.json per region

# --- styled Excel deliverable ---
python3 build_deliverable.py       # -> coverage_gaps_by_region.xlsx

To run against your own data instead of the sample, drop a reference workbook and one *_pipeline.xlsx per region into ./input/ and run python3 extract.py first. Paths are configurable via environment variables (see .env.example): INPUT_DIR, REFERENCE_XLSX, PIPELINE_GLOB, WF_DIR.

Data & privacy

This repository contains code only. It ships no reference registry, no scraped pipeline, and no generated report — only a small synthetic sample (sample/data.json) with invented project names, purely to demonstrate the data shapes and the matching rules. Bring your own Excel inputs and the pipeline reconstructs everything locally. The generated data.json, fuzzy.json, wf_args.json, wf/ and *.xlsx artifacts are git-ignored.

Project layout

common.py             normalisation + fuzzy scorer + region pairing
extract.py            Excel inputs        -> data.json
fuzzy.py              rapidfuzz hints     -> fuzzy.json
build_args.py         merge + pair        -> wf_args.json
split_regions.py      per-region split    -> wf/region_*.json + manifest
coverage.wf.js        adversarial multi-agent verification workflow
build_deliverable.py  results             -> coverage_gaps_by_region.xlsx
sample/data.json      synthetic demo input

License

MIT — see LICENSE.