Skip to content

Add Belgian exchange-rate and pension sources for Microcosm-BE transport - #385

Merged
MaxGhenis merged 7 commits into
mainfrom
be/transport-sources-fx-pensions
Oct 24, 2026
Merged

MaxGhenis merged 7 commits into
mainfrom
be/transport-sources-fx-pensions

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

This PR adds three Belgian sources for Microcosm-BE's transport port (PolicyEngine/microcosm#1369, decision d048).

Inputs they feed (method card BE-MC3, pending Max's ruling d1902):

  • The USD→EUR conversion of donor money.
  • The quantile map of legal pensions onto Belgian amount bands.
  • Pension caseload targets by region and commune.

Each package is a source-only facts package. Every published cell becomes one fact, and nothing is derived, rescaled or aged.

Packages

Package Facts Source
eurostat-ert-bil-eur-a-usd (new) 55 Eurostat ert_bil_eur_a: annual average USD per euro/ECU, 1971–2025. 2024 is 1.0824.
sfpd-pension-amount-bands-2025 (new) 10,239 FPD Jaarlijkse statistiek 2025, sections 1.2, 6.2 and 9: January 2025 recipients and monthly expenditure by gross amount band, sex, career, category, age and IGO eligibility.
sfpd-legal-pension-caseload-2025 (extended) 20,954 PensionStat 2025 workbook (CC0). Pensioners, commune caseloads, IGO counts, averages and medians, printed annual expenditure, and minimum pensions. These replace the package's four curated webpage headlines, which are not cells in the workbook.

GPT-6.1 Sol lanes built each package. Each lane passed validate-package, build-suite and focused tests, plus independent ledger-source-fidelity and ledger-boundary reviews. The pension-band lane also re-read the PDF with Poppler and matched all 10,239 values and locators.

Changes made at integration

PensionStat communes. The lane matched 511 of 565 commune names and kept 54 as publisher-local scopes. Those 54 were mostly Dutch names of Walloon and Brussels communes: Aarlen, Luik, Elsene and so on.

Matching against Statbel's official Dutch and French names (now on main through #383) and the publisher's own French edition assigns all 565. The generator fails unless the result is a bijection onto Statbel's 565 NIS 2025 communes. The tiers:

  • Statbel Dutch name: 551 exact and 9 orthographic (case, diacritics, apostrophe style, œ).
  • French edition: 4 communes. Each commune of the Dutch edition is paired with the French-edition row that has the identical 2019–2025 count series; all 565 series are distinct. The French label then matches Statbel's French name. This resolves Belle (Belœil), Gembloers (Gembloux), Heilissem (Hélécine) and 's-Gravenbrakel (Braine-le-Comte).
    • The French workbook is registered as label evidence (label_evidence_fr) and yields no facts.
  • One declared section label: PensionStat calls Saint-Nicolas (Liège province, 62093) by its section Montegnée in both editions, because its French edition labels Sint-Niklaas "Saint-Nicolas".
    • The generator declares that collision and fails if it stops holding.
    • 62093 is also the only commune left without a row once the other 564 are assigned.

commune_mapping.json records each commune's tier, its French-edition label and cell, Statbel's names and the source cell.

Geography names. Country, NUTS1 and commune names follow the existing Statbel packages ("Belgium", "Flemish Region", "Arlon"). The publisher's labels stay in each package's own dimensions. The lanes had used "België" and the Dutch region names, which would have added four cross-package name warnings.

Period coverage of counts (from the first review). Both publishers date some counts to a stock on 1 January, not to the month:

  • The FPD tables' section headers read "Toestand op 1 januari 2025" (PDF pages 25, 225 and 239).
  • PensionStat's IGO page says "De aantallen hebben betrekking op 1 januari 2025".

So the 5,572 FPD recipient counts and 1,715 IGO counts now have coverage from 1 January 2025 to 1 January 2025. Monthly expenditure and the IGO mean and median stay at 1–31 January.

The other PensionStat sheets keep January coverage, because their pages count everyone who received a pension in January 2025. Those sheets are pensioners, communes and minimum pensions.

Coverage is declared per record set, so:

  • each FPD band group is now one record set per metric;
  • the IGO sheet is two record sets, counts and amounts.

The split sets share their table's spec ID and source-record-ID prefix. All 31,193 fact keys, values, periods and source-record IDs are therefore unchanged, and so is every bundle pin.

What moves, for the 15,384 facts in the split tables:

  • layout.record_set_id and the layout spec hash;
  • the measure ordinal, for the 8,097 facts whose measure moved into a new set (FPD expenditure, IGO mean and median);
  • the coverage notes;
  • the coverage end date and source label, for the 7,287 counts only.

Rebase onto main. The lanes branched from 8f4cfd5, before #383 and #384. All bundle pins were re-derived on 200bf6d. Notably, chronicle#381 already added calendar years 1991–1994, so the exchange-rate series adds 18 period keys, not 22.

Merged-bundle pins

The merged bundle changes as follows:

  • Facts: 1,586,258 → 1,617,502 (+55 +10,239 +20,954 −4 retired).
  • Sources: 57 → 58.
  • Packages: 413 → 415.
  • Tables: 355 → 357. The PensionStat workbook table replaces the curated one.
  • Geographies: 102,613 → 102,615 (Eurostat's USD scope and PensionStat's residents abroad, BUI). The 565 commune keys already exist through Add Statbel commune population facts for NIS 2025 #383.
  • Periods: 535 → 553.
  • Warnings: 123 (+1). Eurostat's exchange-rate dataset labels freq=A "Annual", as its JSON-stat file prints it, where the seven older Eurostat packages carry the bare code A. That adds one conflicting_value_label_across_packages warning, pinned in 49809b6. The first CI bundle run at 0a6f053 caught it; it was the only failing assertion.

Per-key pins are updated for country:BE, nuts1:BE1..3, commune:11002, entities, sources and periods. New source-only coverage tests in tests/test_chronicle_bundle.py record the measurement behind each pin.

Invariants (tested)

  • One fact per published cell. Every numeric 2025 cell in the selected sheets, sections and series becomes exactly one fact. Values are verbatim, with no scaling, ratios or totals.
  • Communes. PensionStat's commune rows map onto NIS 2025 as a bijection (565 → 565). Geography names equal the Statbel canonical names.
  • Determinism. Each generator reproduces its checked YAML byte for byte, with no YAML anchors or merge keys.
  • Coverage follows the publisher's reference date. Counts dated "Toestand op 1 januari 2025" or "betrekking op 1 januari 2025" cover that day. Monthly amounts cover January. Split record sets keep one table identity.
  • Raw captures. Each capture's SHA-256 and size match its manifest. R2 keys are content-addressed under ledger-raw.

Checks

Check Result
test_chronicle_pensionstat.py 12 passed locally at f65e629; passed at 0a6f053 in CI (Chronicle surface)
test_chronicle_sfpd_pension_amount_bands.py 20 passed locally at f65e629; passed at 0a6f053 in CI (Chronicle surface)
test_source_eurostat_exchange_rates.py 13 passed
test_belgium_targets.py 26 passed
test_chronicle_bundle.py -k source_coverage 2 passed
Fact identities before and after 0a6f053 (all 31,193 facts) keys, values, periods and source-record IDs identical; coverage dates changed for exactly the 7,287 counts
validate-package / build-suite, each package valid, 0 errors, 0 warnings
Per-source bundle checks (dimension labels, geography names) 0 issues
ruff check chronicle db scripts tests (0.14.10) pass
CI Chronicle surface at 0a6f053 (every test file except the bundle, UK national and OpenTimestamps files) pass

The merged-bundle test was not run locally (about 15 GB peak, about 2.5 hours). CI's bundle job is its measurement. The test stops at its first failed assertion, so the 0a6f053 run never reached its later pins. Those were checked by replaying the test against that run's uploaded bundle reports: the replay fails at 0a6f053 exactly where CI did, catches deliberately broken pins, and passes at 49809b6.

All five raw files are in R2 ledger-raw under their content-addressed keys, each read back and matched by SHA-256.

Review. The independent GPT-6.1 Sol review of f65e629 confirmed every bundle pin, the commune bijection and all sampled cells. It requested the two coverage fixes above.

Its re-review of 0a6f053 approved with no findings. An independent Opus 5.5 review of the 49809b6 pin commit also approved; it checked the warning against all eight Eurostat raw files and the artifact. That commit also sorts an import, a cosmetic change: the repo's ruff config does not select the import-order rule, despite its commit message. It checked each sheet's coverage against the publisher's definition pages, and independently rebuilt both revisions' facts: keys, values, periods, measures, source-record IDs and lineage are identical for all 31,193 facts.

🤖 Generated with Claude Code

MaxGhenis and others added 5 commits October 23, 2026 18:43
Preserve all 55 published annual cells for 1971-2025 from the immutable
Eurostat JSON-stat capture, including 2024 at 1.0824 USD per EUR. Add a
deterministic generator, guarded source-cell lineage, declared R2 storage
and independent source-value tests.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preserve 10,239 published cells from sections 1.2, 6.2 and 9 of the
Jaarlijkse statistiek 2025, with exact PDF locators, population
distinctions, the immutable PDF capture and R2 declarations. Add
deterministic generation and independent PDF checks. The country
geography is named Belgium, as in the Statbel packages.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the four curated headlines with direct, guarded selectors for
20,954 published cells across pensioners, commune caseloads, IGO,
expenditure and minimum pensions. Preserve the raw workbook, native
precision and reference periods, and declare R2 keys.

Map all 565 commune names to NIS 2025 codes as a checked bijection:
Statbel's official Dutch names resolve 560, the publisher's French
edition (registered as label evidence, paired by identical 2019-2025
count series) resolves four through Statbel's French names, and one
declared section label maps Montegnee to Saint-Nicolas (62093). Commune
and region geography names follow the Statbel packages; the publisher's
labels stay in the pensionstat dimensions.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three packages add 31,244 facts net (55 + 10,239 + 20,954, less
PensionStat's four retired headlines), one source, two packages, two
tables net, two statistical-scope geographies and 18 calendar years
(1972-1980 and 1982-1990; chronicle#381 already added 1991-1994). Their
country, NUTS1 and commune names match the existing Statbel packages, so
no geography-name warning is added. Source-only coverage tests record
the measurements behind each pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The FPD amount-band tables date every selected population "Toestand op
1 januari 2025" (printed pp15, 215, 229), and PensionStat's IGO page dates
its counts to 1 January 2025. Both packages gave those counts whole-month
January coverage. Counts now cover 1 January 2025 only; monthly amounts
keep January.

Coverage is declared per record set, so each band group splits into one
record set per metric and the IGO sheet into counts and amounts. Split
sets share the table's spec ID and source-record-ID prefix: all 31,193
fact keys, values, periods and source-record IDs are unchanged, and only
coverage (7,287 counts) and record-set IDs move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Review of f65e629 (GPT-6.1 Sol, REQUEST_CHANGES with two P2 findings), addressed in 0a6f053:

  1. FPD recipient counts dated to 1 January. The 5,572 FPD recipient counts now cover 2025-01-01 to 2025-01-01, matching "Toestand op 1 januari 2025" on PDF pages 25, 225 and 239. Monthly expenditure keeps January.
  2. IGO counts dated to 1 January. The 1,715 IGO counts likewise cover 2025-01-01 to 2025-01-01, matching "De aantallen hebben betrekking op 1 januari 2025". IGO means and medians keep January.

Record sets are split per coverage but share each table's spec ID and source-record-ID prefix. A before/after dump of all 31,193 facts shows identical aggregate, semantic and legacy keys, values, periods and source-record IDs, so the bundle pins are unchanged. I checked the other PensionStat sheets against their definition pages: pensioners, communes and minimum pensions count recipients "in januari 2025", so they keep January coverage.

The tests now restate the publisher dates independently and check the PDF headers. The re-review is running, and the focused suites are queued behind tonight's host reservation.

@MaxGhenis
MaxGhenis marked this pull request as ready for review October 24, 2026 02:24
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merge audit for head 0a6f0538271b87c109136b64c444ff35a9e66f79.

  • Independent review: GPT-6.1 Sol re-review of 0a6f053 approved with no findings. It rebuilt both revisions' facts, and all 31,193 are identical in key, value, period, measure, source-record ID and lineage. The earlier review of f65e629 asked for the two coverage fixes this head contains.
  • Checks: Chronicle surface passed at this head. Chronicle bundle is running. A watcher merges with squash, --match-head-commit 0a6f0538, only when gh pr checks exits 0 and the PR is MERGEABLE, not a draft and has no CHANGES_REQUESTED. It aborts if the head moves.
  • If the only block is the required approving review, the watcher admin-merges at this head under the standing rule for the owner's own PRs. Failing or pending checks and conflicts are never bypassed.
  • Shared code: only SOURCE_PACKAGE_ALIASES registration lines and the bundle pins for these packages.

🤖 Generated with Claude Code

The bundle CI run at 0a6f053 counted 123 warnings against the pinned 122.
The extra one is conflicting_value_label_across_packages for freq=A: the
exchange-rate package keeps Eurostat's JSON-stat category label "Annual",
where the seven older Eurostat packages carry the bare code "A". Chronicle
keeps each publisher's wording and warns, so the pin moves, not the label.

Also sort the consumer_contract import that the transport commit added
out of order (ruff I001).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Audit, 49809b6: the CI bundle run at 0a6f053 failed on one pin, warning_count 123 against the pinned 122.

The extra warning is conflicting_value_label_across_packages for freq=A. Eurostat's exchange-rate dataset labels it "Annual", as its JSON-stat file prints it. The seven older Eurostat packages carry the bare code. 49809b6 pins the warning and leaves both labels as published.

The test stops at its first failed assertion, so CI never reached the later pins. I checked them by replaying the full test against that run's uploaded bundle reports. The replay fails at 0a6f053 where CI did and passes at 49809b6.

Independent review of 49809b6 (Opus 5.5): APPROVE. Its one request, the stale warning count in the PR body, is fixed. Earlier reviews: GPT-6.1 Sol approved 0a6f053 with no findings; that review is quoted in the PR body. This PR merges (squash) only when CI is green at 49809b6.

🤖 Generated with Claude Code

…ort sources

The one conflict is in tests/test_chronicle_bundle.py's source-package
pins. This branch adds built packages, so it keeps source_package_count
at 415. #386's seven county packages have no 2023 artifact and are
skipped, so main's skipped_source_count of 24 and its skipped-source
list are taken. The diff from the reviewed head 49809b6 is exactly
#386's diff from 200bf6d.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged origin/main (#386, ACS county packages) in a8f68f0.

This will squash-merge at a8f68f0 once CI passes on this head.

@MaxGhenis
MaxGhenis merged commit ba2509a into main Oct 24, 2026
3 checks passed
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
The package adds 565 government facts at tax_year:2025, one per existing
commune key, so geography_count is unchanged and commune:11002 goes from
9 to 10. It adds one source, one source table and one built package.
SPF spells nine communes unlike Statbel, and the package keeps SPF's
wording, so nine commune geography-name warnings are added (123 -> 132).

The deltas were measured with Chronicle's own bundle functions. A replay
of the full bundle test against #385's CI bundle, with #386's seven
skipped county packages and this package added, passes, and main's
unchanged test fails against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
…ages

The two packages add 34,881 person facts at country:BE: 34,629 quarterly
daily-wage-class cells from 2023 Q1 to 2026 Q1, and 252 annual key
figures for 2020-2025. They add two sources, two source tables and two
built packages. They add no new geography, period or entity key and no
warning.

This stacks on #387's pins. The deltas were measured with Chronicle's
own bundle functions and confirmed against #388's CI bundle on the old
base. A replay of the full bundle test against #385's CI bundle, with
#386's skipped packages, #387's package and these two packages added,
passes, and #387's test fails against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
The package adds 35 calendar-2025 facts: 19 person, 2 benefit_unit
premium payments and 14 government expenditure. It adds one
statistical_scope geography, the German-speaking Community, plus one
source, one source table and one built package. It adds no warning.

This stacks on #388's pins. The deltas were measured with Chronicle's
own bundle functions and confirmed against #389's CI bundle on the old
base. A replay of the full bundle test against #385's CI bundle, with
#386's skipped packages and #387's, #388's and this package added,
passes, and #388's test fails against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis deleted the be/transport-sources-fx-pensions branch October 24, 2026 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant