Repository navigation
Add Belgian exchange-rate and pension sources for Microcosm-BE transport - #385
Conversation
Preserve all 55 published annual cells for 1971-2025 from the immutable Eurostat JSON-stat capture, including 2024 at 1.0824 USD per EUR. Add a deterministic generator, guarded source-cell lineage, declared R2 storage and independent source-value tests. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preserve 10,239 published cells from sections 1.2, 6.2 and 9 of the Jaarlijkse statistiek 2025, with exact PDF locators, population distinctions, the immutable PDF capture and R2 declarations. Add deterministic generation and independent PDF checks. The country geography is named Belgium, as in the Statbel packages. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the four curated headlines with direct, guarded selectors for 20,954 published cells across pensioners, commune caseloads, IGO, expenditure and minimum pensions. Preserve the raw workbook, native precision and reference periods, and declare R2 keys. Map all 565 commune names to NIS 2025 codes as a checked bijection: Statbel's official Dutch names resolve 560, the publisher's French edition (registered as label evidence, paired by identical 2019-2025 count series) resolves four through Statbel's French names, and one declared section label maps Montegnee to Saint-Nicolas (62093). Commune and region geography names follow the Statbel packages; the publisher's labels stay in the pensionstat dimensions. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three packages add 31,244 facts net (55 + 10,239 + 20,954, less PensionStat's four retired headlines), one source, two packages, two tables net, two statistical-scope geographies and 18 calendar years (1972-1980 and 1982-1990; chronicle#381 already added 1991-1994). Their country, NUTS1 and commune names match the existing Statbel packages, so no geography-name warning is added. Source-only coverage tests record the measurements behind each pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The FPD amount-band tables date every selected population "Toestand op 1 januari 2025" (printed pp15, 215, 229), and PensionStat's IGO page dates its counts to 1 January 2025. Both packages gave those counts whole-month January coverage. Counts now cover 1 January 2025 only; monthly amounts keep January. Coverage is declared per record set, so each band group splits into one record set per metric and the IGO sheet into counts and amounts. Split sets share the table's spec ID and source-record-ID prefix: all 31,193 fact keys, values, periods and source-record IDs are unchanged, and only coverage (7,287 counts) and record-set IDs move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review of f65e629 (GPT-6.1 Sol, REQUEST_CHANGES with two P2 findings), addressed in 0a6f053:
Record sets are split per coverage but share each table's spec ID and source-record-ID prefix. A before/after dump of all 31,193 facts shows identical aggregate, semantic and legacy keys, values, periods and source-record IDs, so the bundle pins are unchanged. I checked the other PensionStat sheets against their definition pages: pensioners, communes and minimum pensions count recipients "in januari 2025", so they keep January coverage. The tests now restate the publisher dates independently and check the PDF headers. The re-review is running, and the focused suites are queued behind tonight's host reservation. |
|
Merge audit for head
🤖 Generated with Claude Code |
The bundle CI run at 0a6f053 counted 123 warnings against the pinned 122. The extra one is conflicting_value_label_across_packages for freq=A: the exchange-rate package keeps Eurostat's JSON-stat category label "Annual", where the seven older Eurostat packages carry the bare code "A". Chronicle keeps each publisher's wording and warns, so the pin moves, not the label. Also sort the consumer_contract import that the transport commit added out of order (ruff I001). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Audit, 49809b6: the CI bundle run at 0a6f053 failed on one pin, The extra warning is The test stops at its first failed assertion, so CI never reached the later pins. I checked them by replaying the full test against that run's uploaded bundle reports. The replay fails at 0a6f053 where CI did and passes at 49809b6. Independent review of 49809b6 (Opus 5.5): APPROVE. Its one request, the stale warning count in the PR body, is fixed. Earlier reviews: GPT-6.1 Sol approved 0a6f053 with no findings; that review is quoted in the PR body. This PR merges (squash) only when CI is green at 49809b6. 🤖 Generated with Claude Code |
…ort sources The one conflict is in tests/test_chronicle_bundle.py's source-package pins. This branch adds built packages, so it keeps source_package_count at 415. #386's seven county packages have no 2023 artifact and are skipped, so main's skipped_source_count of 24 and its skipped-source list are taken. The diff from the reviewed head 49809b6 is exactly #386's diff from 200bf6d. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged origin/main (#386, ACS county packages) in a8f68f0.
This will squash-merge at a8f68f0 once CI passes on this head. |
The package adds 565 government facts at tax_year:2025, one per existing commune key, so geography_count is unchanged and commune:11002 goes from 9 to 10. It adds one source, one source table and one built package. SPF spells nine communes unlike Statbel, and the package keeps SPF's wording, so nine commune geography-name warnings are added (123 -> 132). The deltas were measured with Chronicle's own bundle functions. A replay of the full bundle test against #385's CI bundle, with #386's seven skipped county packages and this package added, passes, and main's unchanged test fails against it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ages The two packages add 34,881 person facts at country:BE: 34,629 quarterly daily-wage-class cells from 2023 Q1 to 2026 Q1, and 252 annual key figures for 2020-2025. They add two sources, two source tables and two built packages. They add no new geography, period or entity key and no warning. This stacks on #387's pins. The deltas were measured with Chronicle's own bundle functions and confirmed against #388's CI bundle on the old base. A replay of the full bundle test against #385's CI bundle, with #386's skipped packages, #387's package and these two packages added, passes, and #387's test fails against it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The package adds 35 calendar-2025 facts: 19 person, 2 benefit_unit premium payments and 14 government expenditure. It adds one statistical_scope geography, the German-speaking Community, plus one source, one source table and one built package. It adds no warning. This stacks on #388's pins. The deltas were measured with Chronicle's own bundle functions and confirmed against #389's CI bundle on the old base. A replay of the full bundle test against #385's CI bundle, with #386's skipped packages and #387's, #388's and this package added, passes, and #388's test fails against it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR adds three Belgian sources for Microcosm-BE's transport port (PolicyEngine/microcosm#1369, decision d048).
Inputs they feed (method card BE-MC3, pending Max's ruling d1902):
Each package is a source-only facts package. Every published cell becomes one fact, and nothing is derived, rescaled or aged.
Packages
eurostat-ert-bil-eur-a-usd(new)ert_bil_eur_a: annual average USD per euro/ECU, 1971–2025. 2024 is 1.0824.sfpd-pension-amount-bands-2025(new)sfpd-legal-pension-caseload-2025(extended)GPT-6.1 Sol lanes built each package. Each lane passed
validate-package,build-suiteand focused tests, plus independentledger-source-fidelityandledger-boundaryreviews. The pension-band lane also re-read the PDF with Poppler and matched all 10,239 values and locators.Changes made at integration
PensionStat communes. The lane matched 511 of 565 commune names and kept 54 as publisher-local scopes. Those 54 were mostly Dutch names of Walloon and Brussels communes: Aarlen, Luik, Elsene and so on.
Matching against Statbel's official Dutch and French names (now on main through #383) and the publisher's own French edition assigns all 565. The generator fails unless the result is a bijection onto Statbel's 565 NIS 2025 communes. The tiers:
œ).label_evidence_fr) and yields no facts.commune_mapping.jsonrecords each commune's tier, its French-edition label and cell, Statbel's names and the source cell.Geography names. Country, NUTS1 and commune names follow the existing Statbel packages ("Belgium", "Flemish Region", "Arlon"). The publisher's labels stay in each package's own dimensions. The lanes had used "België" and the Dutch region names, which would have added four cross-package name warnings.
Period coverage of counts (from the first review). Both publishers date some counts to a stock on 1 January, not to the month:
So the 5,572 FPD recipient counts and 1,715 IGO counts now have coverage from 1 January 2025 to 1 January 2025. Monthly expenditure and the IGO mean and median stay at 1–31 January.
The other PensionStat sheets keep January coverage, because their pages count everyone who received a pension in January 2025. Those sheets are pensioners, communes and minimum pensions.
Coverage is declared per record set, so:
The split sets share their table's spec ID and source-record-ID prefix. All 31,193 fact keys, values, periods and source-record IDs are therefore unchanged, and so is every bundle pin.
What moves, for the 15,384 facts in the split tables:
layout.record_set_idand the layout spec hash;Rebase onto main. The lanes branched from 8f4cfd5, before #383 and #384. All bundle pins were re-derived on 200bf6d. Notably, chronicle#381 already added calendar years 1991–1994, so the exchange-rate series adds 18 period keys, not 22.
Merged-bundle pins
The merged bundle changes as follows:
BUI). The 565 commune keys already exist through Add Statbel commune population facts for NIS 2025 #383.freq=A"Annual", as its JSON-stat file prints it, where the seven older Eurostat packages carry the bare codeA. That adds oneconflicting_value_label_across_packageswarning, pinned in 49809b6. The first CI bundle run at 0a6f053 caught it; it was the only failing assertion.Per-key pins are updated for
country:BE,nuts1:BE1..3,commune:11002, entities, sources and periods. New source-only coverage tests intests/test_chronicle_bundle.pyrecord the measurement behind each pin.Invariants (tested)
ledger-raw.Checks
test_chronicle_pensionstat.pytest_chronicle_sfpd_pension_amount_bands.pytest_source_eurostat_exchange_rates.pytest_belgium_targets.pytest_chronicle_bundle.py -k source_coveragevalidate-package/build-suite, each packageruff check chronicle db scripts tests(0.14.10)The merged-bundle test was not run locally (about 15 GB peak, about 2.5 hours). CI's bundle job is its measurement. The test stops at its first failed assertion, so the 0a6f053 run never reached its later pins. Those were checked by replaying the test against that run's uploaded bundle reports: the replay fails at 0a6f053 exactly where CI did, catches deliberately broken pins, and passes at 49809b6.
All five raw files are in R2
ledger-rawunder their content-addressed keys, each read back and matched by SHA-256.Review. The independent GPT-6.1 Sol review of f65e629 confirmed every bundle pin, the commune bijection and all sampled cells. It requested the two coverage fixes above.
Its re-review of 0a6f053 approved with no findings. An independent Opus 5.5 review of the 49809b6 pin commit also approved; it checked the warning against all eight Eurostat raw files and the artifact. That commit also sorts an import, a cosmetic change: the repo's ruff config does not select the import-order rule, despite its commit message. It checked each sheet's coverage against the publisher's definition pages, and independently rebuilt both revisions' facts: keys, values, periods, measures, source-record IDs and lineage are identical for all 31,193 facts.
🤖 Generated with Claude Code