Skip to content

Add ACS 2020–2024 county income source packages - #386

Merged
MaxGhenis merged 5 commits into
mainfrom
acs-5yr-county-income-2024
Oct 24, 2026
Merged

MaxGhenis merged 5 commits into
mainfrom
acs-5yr-county-income-2024

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Adds seven Census ACS 2020–2024 5-year county source packages, with publisher estimates and margins of error kept as separate facts. The 2024 Census API dataset exists, so no 2023 fallback was needed.

Status: implementation and validation complete; retained as a draft for maintainer review. Three commits pin the publisher artifacts, add exact source-column bound validation, and add the generated packages and county tests.

The supplied 2026-10-24 publisher JSON files were copied unchanged, with SHA-256 pins in seven data manifests and FETCH-MANIFEST.json. No county responses were re-fetched. Source URLs contain no API credentials. R2 fields are content-addressed locators; this PR does not assert that bytes were uploaded.

Table Source counties, excluding PR Numeric estimate cells Numeric MOE cells
B01003 3,144 3,144 130
B11001 3,144 9,432 9,432
B19001 3,144 53,448 53,448
B19013 3,144 3,143 3,143
B19025 3,144 3,144 3,144
B19051 3,144 9,432 9,432
B19061 3,144 3,144 3,144

All seven publisher responses contain 3,222 county rows including Puerto Rico. Full source rows preserve those responses; facts exclude state 72. County GEO_ID and Census NAME are preserved. B11001 includes total, family, and nonfamily households. B19001 includes its total and all 16 brackets, with income bounds in fact constraints and package metadata; the first and last brackets have unbounded endpoints.

Publisher special cases:

  • B01003 has 3,014 MOE codes -555555555 with annotation ***** (MOE not appropriate for a controlled estimate). They remain in source rows/cells and unavailable-cell metadata, and are not replaced with zero.
  • B19013 suppresses De Baca County, New Mexico (35011): estimate -666666666 / annotation -, MOE -222222222 / annotation **. Those cells are preserved, with no numeric estimate or MOE fact invented. Therefore the table covers 3,144 source counties and 3,143 numeric counties.
  • No table failed ingestion. There are 166,760 numeric facts and 3,016 preserved unavailable cells.

Verified properties:

  • Each source table covers all 3,144 county-equivalents in the 50 states and DC; Puerto Rico is absent from facts.
  • Connecticut uses exactly its nine planning regions 09110–09190, with no legacy county codes.
  • Each county’s 16 B19001 brackets sum to its B19001 total.
  • B19051 with earnings plus no earnings equals its total; B11001 family plus nonfamily equals its total.
  • Every emitted numeric estimate/MOE is nonnegative and equals its exact publisher cell, with row/cell lineage and pinned artifact provenance.
  • Facts retain end-year 2024 identity plus explicit 2020-01-01–2024-12-31 survey reference coverage, vintage acs_5_year_2020_2024, and publisher 2024-dollar income definitions.
  • No aging, imputation, cross-source reconciliation, consumer target selection, or solver construction is introduced.

The generator reuses the existing full JSON-table parser. The small acceptance-validator change recognizes exact typed lower/upper bounds declared on guarded wide-table columns, matching existing row-bound evidence. It does not change schemas, fact values, identities, or periods. MOE concepts and statistic constraints distinguish them from estimates; metadata explicitly prohibits adding MOEs across categories/counties.

axiom: n/a: source data package

Approved Chronicle agent role: ledger-source-ingestor.

Deterministic checks and tests:

  • python -m chronicle.sources.census_acs5_county --check: all seven packages reproduce byte-for-byte.
  • County tests: 61 passed, serial, including source-package/core/consumer/agent acceptance, exhaustive numeric cell preservation, county inventories, identities, bounds, suppression, unsupported years, and malformed-source rejection.
  • Source-suite tests: 80 passed.
  • Bundle regressions: 24 passed; the full merged-bundle test skipped locally because unrelated R2-only artifacts are unavailable.
  • Broader seven-file run with -n 3: 260 passed, one unrelated LIHEAP PDF cell-count failure with newer unpinned pypdf 6.21.2. Both LIHEAP cases passed when rerun with the repository’s frozen dependencies (pypdf 6.11.0); no unrelated code was changed.
  • Repository-wide Ruff check passed using the lockfile version 0.14.10; diff checks passed.

LLM judge verdicts:

  • ledger-source-fidelity: PASS. Independent exhaustive reconstruction matched all 166,760 numeric declarations and all 3,016 unavailable encodings to publisher bytes, hashes, names/GEOIDs, bounds, and period/vintage.
  • ledger-boundary: PASS. Independent review includes the source-column bound validator; no computed or consumer-owned values are introduced.
  • ledger-contract: n/a; no contract/schema changes.

MaxGhenis and others added 5 commits October 23, 2026 21:18
…for 2023

source_package_count counts built packages only; the seven ACS county
packages have no 2023 artifact and land in skipped_sources (17 -> 24).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ledger-source-fidelity and ledger-boundary judges passed at cc7d6f2
with two findings, both applied here:

- Puerto Rico's 78 municipios are now emitted like every other county row
  (3,222 rows: 3,144 non-PR counties plus 78), matching acs_s0101_state_2024
  and acs_s2201_district_2024. Scope selection belongs to consumers.
  PR's 76 sentinel B01003 MOEs (-555555555 with *****) go to
  unavailable_cells and never become facts.
- B19013's margin of error now carries aggregation "median" like its
  estimate.

All seven source_package.yaml files were regenerated with the generator;
--check passes twice and the raw JSON SHA-256 pins are unchanged. Facts:
166,760 -> 170,896.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Judges at 517ee9d: ledger-source-fidelity PASS, ledger-boundary PASS (Subfleet run 20261023-232551-lwi-j2-chr386). Their one should-fix was that the nine mutation tests couldn't create temp directories in the judge sandbox. I reran tests/test_chronicle_census_acs5_county.py locally with a writable TMPDIR: 65 passed, including all nine mutation tests.

@MaxGhenis
MaxGhenis marked this pull request as ready for review October 24, 2026 05:21
@MaxGhenis
MaxGhenis merged commit b2b5d52 into main Oct 24, 2026
3 checks passed
@MaxGhenis
MaxGhenis deleted the acs-5yr-county-income-2024 branch October 24, 2026 06:03
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
…ort sources

The one conflict is in tests/test_chronicle_bundle.py's source-package
pins. This branch adds built packages, so it keeps source_package_count
at 415. #386's seven county packages have no 2023 artifact and are
skipped, so main's skipped_source_count of 24 and its skipped-source
list are taken. The diff from the reviewed head 49809b6 is exactly
#386's diff from 200bf6d.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
The package adds 565 government facts at tax_year:2025, one per existing
commune key, so geography_count is unchanged and commune:11002 goes from
9 to 10. It adds one source, one source table and one built package.
SPF spells nine communes unlike Statbel, and the package keeps SPF's
wording, so nine commune geography-name warnings are added (123 -> 132).

The deltas were measured with Chronicle's own bundle functions. A replay
of the full bundle test against #385's CI bundle, with #386's seven
skipped county packages and this package added, passes, and main's
unchanged test fails against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
…ages

The two packages add 34,881 person facts at country:BE: 34,629 quarterly
daily-wage-class cells from 2023 Q1 to 2026 Q1, and 252 annual key
figures for 2020-2025. They add two sources, two source tables and two
built packages. They add no new geography, period or entity key and no
warning.

This stacks on #387's pins. The deltas were measured with Chronicle's
own bundle functions and confirmed against #388's CI bundle on the old
base. A replay of the full bundle test against #385's CI bundle, with
#386's skipped packages, #387's package and these two packages added,
passes, and #387's test fails against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 24, 2026
The package adds 35 calendar-2025 facts: 19 person, 2 benefit_unit
premium payments and 14 government expenditure. It adds one
statistical_scope geography, the German-speaking Community, plus one
source, one source table and one built package. It adds no warning.

This stacks on #388's pins. The deltas were measured with Chronicle's
own bundle functions and confirmed against #389's CI bundle on the old
base. A replay of the full bundle test against #385's CI bundle, with
#386's skipped packages and #387's, #388's and this package added,
passes, and #388's test fails against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant