CHANGELOG.md

[1.53.1] - 2026-04-20

No significant changes.

[1.53.0] - 2026-04-19

Added

• Tightened test_population tolerance from 7% to 3% now that the stage-2 QRF (#362), TFC target refresh (#363), and reported-anchor takeup (#359) pulled the weighted UK population overshoot from ~6.5% down to ~1.6%. Added four regression tests in test_population_fidelity.py (weighted-total match, household-count range, non-inflation guard, country-sum consistency) extracted from the earlier #310 draft so any future calibration drift back toward the pre-April-2026 overshoot trips CI.

[1.52.2] - 2026-04-18

Changed

• Add second-stage QRF imputation of FRS-only variables on SPI-donor rows. After the first-stage SPI-trained QRF overwrites income components on the zero-weight subsample, a new second-stage QRF trained on the full FRS rewrites benefit _reported columns, pension contributions, and savings-income so they correlate with the freshly-imputed incomes instead of staying as whatever middle-income FRS donor was sampled. Mirrors the policyengine-us-data#589 pattern. Prevents synthetic £2 M earners from carrying a middle-income donor's UC / housing-benefit receipt into calibration, which was blowing up benefit aggregates under upweight.
• Anchor stochastic takeup assignment for Universal Credit, Pension Credit, and Child Benefit to the FRS-reported receipt columns, matching the policyengine-us-data pattern. Respondents who report positive receipt in the FRS benefits table now receive would_claim_* = True with certainty, and non-reporters are filled probabilistically to hit the aggregate target rate. Removes a source of calibration noise where respondents who clearly took up a benefit could be randomly assigned would_claim = False.

Fixed

• Refresh Tax-Free Childcare calibration targets and take-up rate using HMRC's June 2025 release (covering 2024-25 outturn: £632 m spending, 985 k children reached). The prior target set was calibrated against the September 2024 release and undershot current TFC spending by roughly a third. Bumps the default TFC take-up rate from 0.586 to 0.88 on 2024-04-06 to close most of the gap pending a full recalibration run.

[1.52.1] - 2026-04-18

Fixed

• Update the Raise VAT standard rate by 2pp reform-impact test expectation from 25.0 bn to 43.0 bn — the enhanced FRS's total consumption aggregate has grown to a UK-realistic ~£1.6 T (matching ONS 2025 total consumer expenditure), so a 2pp rise on the current microdata_vat_coverage = 0.38-scaled base produces ~£43 bn, not the £25 bn calibrated against an older smaller dataset. Also clamps raw electricity/gas consumption in impute_energy_splits to be non-negative (a handful of LCFS bill-variable inconsistencies produced small negatives), fixing test_non_negative_energy. Follow-up: revisit microdata_vat_coverage itself now that the underlying base is fuller (#364).

[1.52.0] - 2026-04-17

Changed

• Point CONTRIBUTING.md at the shared PolicyEngine contribution guide (https://github.com/PolicyEngine/.github) and trim the per-repo file to commands, repo-specific conventions, and anti-patterns. Removes the stale changelog_entry.yaml / make changelog instructions.

Removed

• Remove policyengine_uk_data/tests/test_changelog_encoding.py. It validated UTF-8 and YAML structure of the deprecated changelog_entry.yaml, which was retired when the repo migrated to towncrier changelog.d/ fragments. All three tests now unconditionally pytest.skip because the file no longer exists, and any fragment-format validation is already handled by the Check changelog fragment CI step.

[1.51.1] - 2026-04-17

Fixed

• Guard the rent/mortgage rescaling in impute_over_incomes against ZeroDivisionError when the seed dataset's imputation columns sum to zero (e.g. the zero-weight synthetic copy in impute_income).

[1.51.0] - 2026-04-17

Added

• Add policyengine_uk_data.utils.hf_destinations with PRIVATE_REPO / PUBLIC_REPO constants and an AST-based xfail test (tests/test_hf_destinations.py) that flags every upload(...), upload_file(...), upload_files_to_hf(...), and upload_data_files(...) call site that still bypasses the shared constants.
• Add policyengine_uk_data.utils.calibrate.load_weights, a defensive loader that normalises calibration weights to 2D (n_areas, n_records) and validates expected shapes so consumers can't silently read the wrong axis layout across the L2 and L0 calibrators.

Fixed

• Fix calibrate_local_areas non-verbose branch silently failing to save weights because the if epoch % 10 == 0 save block was indented outside the training loop.
• Replace bare * 52 weekly-to-annual conversion in LCFS imputation with the shared WEEKS_IN_YEAR = 365.25 / 7 constant used by datasets/frs.py, and replace two np.random.seed(42) calls with local np.random.default_rng(42) so consumption imputation stops mutating the process-wide RNG state.
• Add OBR calibration targets for NIC Classes 2, 3 and 4 (self-employed flat-rate, voluntary and profit-based) alongside the existing Class 1 employee/employer rows, and accept common label-wording variants in OBR EFO Table 3.4.
• Fix datasets/spi.py __main__ crash (two-arg call to three-arg create_spi), parameterise the hardcoded £1,250 marriage allowance from policyengine-uk parameters, seed the age imputation RNG, and surface unknown GORCODE regions as UNKNOWN instead of silently mapping them to SOUTH_EAST.
• Raise UpratingYearOutOfRangeError with a clear message when uprate_values or uprate_dataset is called with a year outside the [START_YEAR, END_YEAR] range of the uprating factor table, instead of surfacing a pandas KeyError or silently returning wrong values.
• Parameterise the VAT standard rate and reduced-rate share in ETB-based VAT imputation by reading from policyengine_uk.parameters.gov.hmrc.vat keyed on the training year, with a VAT_RATE_BY_YEAR fallback for offline use. Promote the etb.year == 2020 filter to a year argument with a DEFAULT_ETB_YEAR default.

[1.50.6] - 2026-04-17

Fixed

• Include gift_aid (SPI GIFTAID) and charitable_investment_gifts (SPI GIFTINV) in the SPI income imputation model so synthetic high-earner rows carry plausible charitable giving drawn jointly with income, instead of a flat zero. Previously the 6-variable QRF ran over only the core income components; both charitable relief columns were in SPI_RENAMES but never reached the predicted output, so the SPI-donor half of the enhanced FRS carried its FRS donor's (always-zero) charitable giving. Adds both columns to the model's output list, renames the cache file to force retraining, and initialises the FRS-side columns to zero to keep the stacked dataset valid.

[1.50.5] - 2026-04-17

No significant changes.

[1.50.4] - 2026-04-17

Fixed

• Add periodic CI heartbeats around long calibration setup stages so dataset release builds do not get canceled while constituency target matrices are being prepared.

[1.50.3] - 2026-04-16

Fixed

• Reduce OA cloning in GitHub Actions dataset builds so publication completes on hosted runners.

[1.50.2] - 2026-04-15

No significant changes.

[1.50.1] - 2026-04-15

No significant changes.

[1.50.0] - 2026-04-15

Added

• Add SLC calibration targets for Parents' Learning Allowance and Adult Dependants' Grant.

[1.49.0] - 2026-04-14

Added

• Add Student Loans Company maintenance-loan recipient-count and spend targets for England full-time undergraduates.

[1.48.0] - 2026-04-13

Added

• Impute student_loan_balance from WAS loan aggregates and retrain stale cached wealth models when that new output is missing.

[1.47.0] - 2026-04-13

Added

• Add 2025 Student Loans Company repayment amount calibration targets by UK country and by England repayment plan.

Fixed

• Preserve zero-valued SLC borrower targets so calibration can enforce explicit zero years.

[1.46.6] - 2026-04-13

Fixed

• Preserve zero-valued SLC borrower targets so calibration can enforce explicit zero years.

[1.46.5] - 2026-04-13

No significant changes.

[1.46.4] - 2026-04-13

Fixed

• Make long-running dataset builds emit plain CI heartbeat logs so release workflows are less likely to die silently during calibration.

[1.46.3] - 2026-04-12

Changed

• Refactor land calibration targets to share one annual ONS land-value series across national targets, regional targets, and tests, including 2021 to 2023 backfill values.

[1.46.2] - 2026-04-12

Changed

• Record build-time model version, git SHA, and data-build fingerprint in UK data release manifests.

[1.46.1] - 2026-04-12

Changed

• Read local-authority ONS income uprating assumptions from policyengine-uk parameters when available, while keeping a compatibility fallback for the current released dependency.

[1.46.0] - 2026-04-12

Added

• Add Output Area crosswalk and geographic assignment for OA-level calibration pipeline.

[1.45.8] - 2026-03-31

No significant changes.

[1.45.7] - 2026-03-31

No significant changes.

[1.45.6] - 2026-03-31

No significant changes.

[1.45.5] - 2026-03-27

Fixed

• Add regional household land value calibration targets (#314).

[1.45.4] - 2026-03-20

Fixed

• Scale property income targets by 1.9x to match HMRC Property Rental Income Statistics, and fix double-counting of aggregate rows in projected years (#230).

[1.45.3] - 2026-03-20

Fixed

• Loosen first decile tax rate threshold from 175% to 200% to accommodate normal variation in low-income decile effective rates.

[1.45.2] - 2026-03-19

Fixed

• Remove the remaining non-land calibration xfail markers and make pytest xfail strict so temporary expected failures cannot silently linger.

[1.45.1] - 2026-03-19

Fixed

• Remove the incorrect property_wealth land calibration target and keep only direct land-value targets from the ONS National Balance Sheet.

[1.45.0] - 2026-03-18

Added

• Added policyengine-claude plugin auto-install configuration.

[1.44.4] - 2026-03-18

Changed

• Clarified that NOMIS ASHE employment income targets cover all workers (part-time and full-time), not just full-time.

[1.44.3] - 2026-03-18

Changed

• Updated age data pipeline to read from raw_age.csv and write processed output to age.csv, preserving original data. Documented original sources in READMEs.

[1.44.2] - 2026-03-18

Fixed

• Fixed ESA calibration target entity mapping: use household_from_family() for the benunit-level esa_income variable instead of household_from_person().

[1.44.1] - 2026-03-18

Changed

• Fixed text errors in methodology documentation page.

[1.44.0] - 2026-03-17

Added

• Add ONS land value calibration targets (household, corporate, total land, property wealth) from the National Balance Sheet 2025.

Fixed

• Fixed versioning workflow by replacing expired PAT with GitHub App token.

[1.43.0] - 2026-03-13

Added

• Upload tiny FRS datasets (frs_2023_24_tiny.h5 and enhanced_frs_2023_24_tiny.h5) to HuggingFace in the CI pipeline.

[1.42.1] - 2026-03-13

Fixed

• Use gross household income (LCFS P344p / FRS household_gross_income) instead of HBAI net income when assigning households to NEED 2023 income bands for energy consumption calibration. NEED uses Experian modelled gross income, so the previous use of net income misallocated households across bands.

[1.42.0] - 2026-03-12

Added

• Added tiny (n=1000 household) versions of frs_2023_24 and enhanced_frs_2023_24 datasets for faster testing and development.

[1.41.0] - 2026-03-06

Added

• Improve electricity and gas imputations to match NEED 2023 admin data across income, tenure, accommodation type, and region using iterative raking calibration.

[1.40.4] - 2026-03-06

Changed

• Switched code formatter from Black to Ruff.

[1.40.3] - 2026-02-24

Changed

• Migrated from changelog_entry.yaml to towncrier fragments to eliminate merge conflicts.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

1.40.2 - 2026-02-23 17:00:30

Fixed

• Revert public HuggingFace upload that would have violated UK Data Service licence terms.

1.40.1 - 2026-02-23 13:26:29

Fixed

• Upload datasets to public HuggingFace repo (policyengine/policyengine-uk-data) in addition to private repo, so policyengine-uk gets the latest data.

1.40.0 - 2026-02-20 13:50:59

Added

• SLC student loan calibration targets for Plan 2 and Plan 5 England borrowers earning above repayment threshold (2025-2030), fetched live from Explore Education Statistics.

1.39.3 - 2026-02-19 16:15:46

Fixed

• Widened UC taper rate reform test tolerance to 15bn to account for calibration variance.

1.39.2 - 2026-02-19 13:58:30

Added

• UC households by children count (0, 1, 2, 3+) as constituency calibration targets.

1.39.1 - 2026-02-19 11:54:03

Added

• Test for highest_education in enhanced FRS dataset.

1.39.0 - 2026-02-19 08:39:08

Added

• highest_education variable derived from FRS EDUCQUAL field.

1.38.0 - 2026-02-18 14:08:40

Changed

• Replaced ad-hoc calibration targets with structured registry and source modules.

1.37.1 - 2026-02-17 16:05:41

Fixed

• Fix salary sacrifice headcount regression caused by uprating mismatch. The calibrator classified records above/below the 2k cap at uprated 2025 prices, but the saved dataset stores values at 2023 prices where the classification differs. Now evaluates SS amounts at base-year prices before applying the threshold, so calibration matches what end-users see. Stage 2 imputation now moves the full employee pension amount to SS (instead of capping at 2k) so both above-cap and below-cap records are created for the calibrator.
• Remove xfail markers from salary sacrifice headcount tests.

1.37.0 - 2026-02-16 10:16:06

Added

• Calibrate salary sacrifice population to HMRC/ASHE headcount targets (7.7mn total, 3.3mn above 2k cap, 4.3mn below 2k cap). Two-stage imputation in salary_sacrifice.py converts pension contributors to below-cap SS users, and three new headcount calibration targets in loss.py.

1.36.2 - 2026-01-21 19:06:42

Fixed

• Use region.values for Scotland comparisons in loss function to ensure consistent behavior with StringArray types

1.36.1 - 2026-01-21 15:14:09

Fixed

• Fix black formatting for 5 files to comply with black 26.1.0

1.36.0 - 2026-01-21 14:22:42

Added

• Add calibration target for UC households in Scotland with youngest child under 1 (~14k from DWP Stat-Xplore)

1.35.1 - 2026-01-17 16:56:01

Changed

• Improved calibration weight initialization to use country-aware divisors. Households now start with weights divided by the number of areas in their country rather than total areas, improving convergence for devolved nation targets.

1.35.0 - 2026-01-17 05:09:55

Added

• Add Scottish Child Payment as a calibration target using Scottish Budget figures

1.34.0 - 2026-01-17 02:11:31

Added

• Add would_claim_scp variable generation for Scottish Child Payment deterministic takeup

1.33.0 - 2026-01-15 16:03:56

Added

• Add calibration target for babies under 1 in Scotland (~46k from NRS Vital Events)

1.32.0 - 2026-01-13 14:24:26

Added

• Scotland-specific calibration targets for children under 16 (NRS mid-year estimates) and households with 3+ children (Census 2022) to improve accuracy of Scottish policy analysis.

1.31.0 - 2026-01-06 10:13:56

Added

• Move all randomness to data package for deterministic country package. Take-up decisions are now generated stochastically during dataset creation using take-up rates from YAML parameter files.

1.30.2 - 2026-01-06 09:55:01

Fixed

• Clip negative values to zero for employment, self-employment, savings interest, tax-free savings, and dividend income variables.

1.30.1 - 2025-12-19 14:41:54

Fixed

• Widened VAT reform test tolerance to accommodate calibration variance between test and production datasets.

1.30.0 - 2025-12-19 11:21:28

Added

• Tests for property_purchased rate and SDLT total validation against official HMRC figures.
• Tests for low-income decile sanity checks to prevent negative net incomes and impossible tax rates.

Fixed

• Set property_purchased stochastically at 3.85% based on HMRC housing transaction data, fixing unrealistic SDLT charges that caused 224% tax rates in the first income decile.

1.29.4 - 2025-12-09 14:39:12

Changed

• Reduced test runtime by using 32 epochs instead of 512 when TESTING environment variable is set

1.29.3 - 2025-12-09 10:06:31

Fixed

• A bug hard-coded to 2023, which meant that custom FRS datasets defined after 2024 fail on BRMA imputation.

1.29.2 - 2025-12-08 17:47:38

Fixed

• Added mortgage and private rent targets.

1.29.1 - 2025-12-07 09:17:37

Fixed

• Zero out fuel spending for households without fuel consumption

1.29.0 - 2025-12-04 17:51:20

Added

• Housing cost calibration to LAs.

1.28.0 - 2025-12-03 23:01:14

Added

• Use num_vehicles as predictor for fuel spending imputation

1.27.1 - 2025-12-03 22:49:46

Changed

• Updated Node.js from 22 to 24 LTS in docs workflow

1.27.0 - 2025-12-03 17:02:06

Added

• Add vehicle ownership calibration and imputation

1.26.1 - 2025-12-02 00:22:31

Fixed

• Made sure LA calibration actually runs.

1.26.0 - 2025-12-01 11:20:56

Fixed

• LA calibration now consistent with constituency calibration.

1.25.0 - 2025-11-29 00:33:46

Added

• Student loan plan imputation based on age and reported repayments

1.24.2 - 2025-11-28 17:23:26

Fixed

• Fix changelog encoding test to skip when changelog_entry.yaml is empty after versioning

1.24.1 - 2025-11-28 16:14:35

Changed

• Calibrate savings income from ONS National Accounts D.41g household interest data instead of SPI (fixes underestimation from ~3bn to ~55bn)

1.24.0 - 2025-11-27 18:33:27

Added

• rail_usage variable derived from rail_subsidy_spending / fare_index at survey year, enabling fare reforms to modify prices independently of usage quantity

1.23.3 - 2025-11-27 17:11:09

Fixed

• Updated personal allowance reform test expected value after calibration fix.

1.23.2 - 2025-11-27 16:29:01

Added

• SS HMRC calibration targets.

1.23.1 - 2025-11-27 16:12:15

Fixed

• Hallucinated calibration targets.

1.23.0 - 2025-11-27 15:30:00

Added

• Add salary sacrifice NI relief as calibration targets (employee £1.2bn, employer £2.9bn from SPP)

1.22.0 - 2025-11-26 22:46:41

Added

• Salary sacrifice imputation using FRS SALSAC routing question to impute ~30% employee participation per HMRC survey data.

1.21.0 - 2025-11-20 13:08:22

Added

• Add pension_contributions_via_salary_sacrifice variable from FRS SPNAMT field

1.20.0 - 2025-10-21 12:09:14

Added

• Universal Credit calibration at national level by award amount and family type, and at constituency level in total.

1.19.6 - 2025-10-21 10:18:38

Changed

• Refactored income imputation to selectively impute only dividend income on the main dataset.
• Removed winter fuel allowance from loss calculations.

1.19.5 - 2025-10-20 16:37:14

Fixed

• Bump patch version to try and get HF upload passing.

1.19.4 - 2025-10-07 16:32:10

Added

• Regional and country labels for UK constituencies.

1.19.3 - 2025-10-02 16:46:59

Changed

• Relaxed childcare test tolerance to allow ratios within 100% of target (0 to 2.0)

1.19.2 - 2025-10-02 16:12:23

Changed

• Relaxed childcare test tolerance to allow ratios up to 1.6

1.19.1 - 2025-10-02 15:18:04

Changed

• Remove birth_year from FRS dataset generation to allow dynamic calculation

1.19.0 - 2025-10-02 14:29:16

Fixed

• Re-add dividends to calibration target set.

1.18.0 - 2025-09-30 13:58:18

Changed

• Upgraded documentation to Jupyter Book 2.0 (MyST-based)

Fixed

• Jupyter Book deployment to GitHub Pages by adding docs workflow and fixing branch reference

1.17.11 - 2025-09-11 16:14:02

Fixed

• Add is_married to FRS benefit unit dataset

1.17.10 - 2025-09-01 09:33:20

Added

• Calibration to benefit cap statistics.

1.17.9 - 2025-08-20 10:32:56

Fixed

• Imputation model syntax.

1.17.8 - 2025-08-06 10:28:59

Fixed

• Test result.

1.17.7 - 2025-08-06 10:08:10

Changed

• Moved to functional, simplified architecture.

1.17.6 - 2025-08-05 11:47:08

Added

• Add index.yaml to fix GitHub Pages 404 error

1.17.5 - 2025-08-05 11:15:17

Added

• Added index.html file to docs folder to fix GitHub Pages 404 error

1.17.4 - 2025-08-04 15:08:06

Fixed

• Migrated to more efficient dataset version.

1.17.3 - 2025-07-22 11:17:39

Fixed

• UK model actually bumped.

1.17.2 - 2025-07-22 09:41:46

Changed

• Updated policyengine-uk to 2.40.2 (pin).

1.17.1 - 2025-07-22 08:53:18

Fixed

• Bug in new multi-year dataset.

1.17.0 - 2025-07-21 13:53:43

Added

• New multi-year dataset format for FRS and Enhanced FRS.

1.16.2 - 2025-07-17 11:44:26

Added

• Council Tax calibration.

1.16.1 - 2025-07-14 15:21:27

Fixed

• Added calibrated weights from 2022.

1.16.0 - 2025-07-10 16:01:46

Added

• Structural insurance payments
• External child payments
• Healthy Start payments

1.15.3 - 2025-07-10 12:41:13

Fixed

• SSMG uprating.

1.15.2 - 2025-07-09 15:25:53

Fixed

• Free school meals magnitude error.

1.15.1 - 2025-07-09 14:19:38

Added

• Free school meals

1.15.0 - 2025-06-27 09:15:13

Added

• PIP calibration.

1.14.4 - 2025-06-24 13:14:48

Fixed

• Missing columns in the calibration log.

1.14.3 - 2025-06-24 11:39:46

Fixed

• Name corrected in calibration build artifact.

1.14.2 - 2025-06-24 11:18:42

Added

• Calibration improvements.

1.14.1 - 2025-06-24 10:48:32

Added

• Calibration log exporting.

1.14.0 - 2025-06-19 21:08:19

Added

• Public service imputations.

1.13.3 - 2025-06-16 15:00:47

Fixed

• Documentation used 2022 datasets rather than 2025.

1.13.2 - 2025-06-13 14:51:39

Fixed

• Documentation publishes.
• Local authority calibration consistent with constituency calibration.
• Domestic rates are nonzero.

1.13.1 - 2025-06-10 12:41:53

Fixed

• Documentation deployment.

1.13.0 - 2025-06-10 12:37:20

Changed

• Tax-benefit targets updated from new DWP forecasts.

1.12.0 - 2025-06-09 20:25:41

Fixed

• Inconsistent local area targets removed.

1.11.6 - 2025-05-27 14:40:08

Fixed

• Uprating in child limit target.

1.11.5 - 2025-05-27 13:30:04

Added

• Child limit affected household calibration.

1.11.4 - 2025-05-26 22:10:54

Fixed

• Added missing HF token

1.11.3 - 2025-05-26 21:44:28

Fixed

• Typo in huggingface repo location.

1.11.2 - 2025-05-26 21:38:53

Fixed

• GCP uploads to buckets happen alongside HF uploads.
• Versioning in data uploads.

1.11.1 - 2024-12-10 10:46:07

Fixed

• Documentation errors.

1.11.0 - 2024-12-09 11:51:05

Added

• Local authority weights.

1.10.1 - 2024-12-03 17:25:35

Added

• Dropout in constituency calibration.

1.10.0 - 2024-12-03 11:21:54

Added

• Target uprating for constituencies.

1.9.2 - 2024-11-30 13:23:17

Fixed

• Constituency weights are in A-Z order.

1.9.1 - 2024-11-27 19:28:29

Added

• Automatic calibration.

1.9.0 - 2024-10-22 11:18:48

Fixed

• Bug removing capital gains.

1.8.0 - 2024-10-22 08:30:52

Changed

• Data URLs updated.

1.7.0 - 2024-10-21 17:03:50

Added

• Calibration for private school students.

1.6.0 - 2024-10-18 16:05:10

Added

• Future year income targeting.
• Random takeup variable values.

1.5.0 - 2024-10-16 17:05:58

Added

• Moved epoch count to 10k per year.

1.4.0 - 2024-10-16 17:05:39

Added

• Missing changelog entry.

1.3.0 - 2024-10-16 17:02:56

Added

• Re-run calibration with more epochs.

1.2.5 - 2024-09-18 13:57:40

Fixed

• GH actions naming.
• Bug causing the Extended FRS to error.

1.2.4 - 2024-09-18 13:26:36

1.2.3 - 2024-09-18 12:57:28

Fixed

• Bug causing the Extended FRS to not generate.

1.2.2 - 2024-09-18 11:38:21

1.2.1 - 2024-09-18 10:05:45

Fixed

• Data download URLs.

1.2.0 - 2024-09-18 00:32:05

Fixed

• Compatibility with PolicyEngine UK.

1.1.0 - 2024-09-17 18:05:27

Changed

• Lightened dependency list.

[1.0.0] - 2024-09-09 17:29:10

Added

• Initialized changelogging