Two-part (zero-inflated) hotspot model with presence gate#2
Closed
sgrabovyi wants to merge 2 commits into
Closed
Conversation
Territories with zero (or a single) recent events in the current window were being misclassified as "Emerging hotspot": the SIR axis compares the current rate to a territory's own history, and when that history is a zero on a large test denominator the Empirical-Bayes baseline collapses to its floor, inflating SIR by construction. Odesa/Kharkiv/Mykolaiv hexes with 0 recent events, and a Poltava hex with a single event, surfaced as hotspots. Adopt the two-part (presence x intensity) decomposition of Fu et al. (Euro Surveill. 2023;28(14):2200581), implemented in our PyMC Beta-Binomial framework: - BayesianAnalyzer.run_two_part_model: zero-inflated Beta-Binomial via pm.CustomDist (marginalised over a latent presence indicator). Presence pi is shared across the current and baseline windows (separate intensities p, p_hist) so it is identified from pooled evidence; the presence-weighted current rate theta = pi * p drives SMR/SIR. Reports presence_prob. Enabled by the two_part_model config flag; routed in bayesian_dispatch. - taxonomy: presence gate. A territory needs at least MIN_HOTSPOT_CURRENT_EVENTS (=2) current recent events, and presence_prob >= 0.5 when available, to receive any increase-hotspot label; otherwise it is Normal. Epidemiologically, a recency hotspot must rest on observed current signal. - excel_report: surface the presence_prob column. - territory_info: deterministic tie-break in the hex -> community/district/oblast labelling (sort by admin id) instead of order-dependent iloc[0], so a hex whose centroid sits on a boundary keeps a stable label across runs. Counts were never affected; the case->hex aggregation is deterministic. - config: enable two_part_model; set analysis_period to 2026-03-01..2026-05-31. Validated on real data (2026-03..05, res4+res3, both models): 0 divergences, zero phantom hotspots, presence_prob discriminates structural zeros (0.69) from present territories (0.85); standard --test passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ship a fully synthetic dataset in the input folder (data/synthetic_input_data.xlsx) that stands in for the confidential case data, and a public notebook (synthetic_data/synthetic_pipeline_public.ipynb) that turns it into the file the pipeline runs on (data/synthetic_input_data_double.xlsx) and demonstrates the geographic-privacy tools (reversible coordinate encryption and donut geomasking) - all with no real data at any point. Add a light data-exploration notebook, a step-by-step getting-started guide (synthetic_data/README.md), and links from README.md and data/README.md. The generation scripts are documented in English. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sgrabovyi
force-pushed
the
two-part-model-zero-handling
branch
from
July 8, 2026 09:34
7a9aba8 to
050b5e3
Compare
Collaborator
Author
|
Closing as subsumed into PR #3, which now targets Everything from this PR is carried forward in the PR #3 branch (it was stacked on top of this one): the two-recent-event presence gate, the deterministic hex→admin labelling fix, and the synthetic test dataset + generator notebook all remain. Only the two-part (zero-inflated) model itself was retired after the biostatistical review — the joint two-period model is now the detector, and zero-count units are handled by the denominator filter plus the presence gate. The net diff in PR #3 reflects that final state. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Territories with zero (or a single) recent events in the current window were
misclassified as Emerging hotspot. The SIR axis compares the current rate to
a territory's own history; when that history is a zero on a large test
denominator, the Empirical-Bayes baseline collapses to its floor and SIR is
inflated by construction. Odesa/Kharkiv/Mykolaiv hexes with 0 recent events, and
a Poltava hex with a single event, surfaced as hotspots.
Approach
Adopt the two-part (presence × intensity) decomposition of Fu et al.
(Euro Surveill. 2023;28(14):2200581), implemented in the existing PyMC
Beta-Binomial framework (no INLA):
run_two_part_model— zero-inflated Beta-Binomial viapm.CustomDist(marginalised over a latent presence indicator). Presence
πis shared acrossthe current and baseline windows (separate intensities
p,p_hist), so it isidentified from pooled evidence; the presence-weighted current rate
θ = π·pdrives SMR/SIR. Reports
presence_prob. Enabled by thetwo_part_modelconfigflag; routed in
bayesian_dispatch.taxonomy) — a territory needs ≥MIN_HOTSPOT_CURRENT_EVENTS(=2) current recent events, and
presence_prob ≥ 0.5when available, to receiveany increase-hotspot label. A recency hotspot must rest on observed current signal.
excel_report— surfacepresence_prob.territory_info— deterministic tie-break in hex → community/district/oblastlabelling (sort by admin id) instead of order-dependent
iloc[0]. Counts areunaffected; the case→hex aggregation is deterministic.
config— enabletwo_part_model; analysis window 2026-03-01..2026-05-31.Validation
Real data (2026-03..05, res4 + res3, both models): 0 divergences, zero phantom
hotspots,
presence_probdiscriminates structural zeros (~0.69) from presentterritories (~0.85); standard
--testpasses.