Skip to content

Improve fireball detector recall with stdpixel decontamination#866

Open
alextudorica wants to merge 1 commit into
CroatianMeteorNetwork:prereleasefrom
alextudorica:fireball-stdpixel-decontamination
Open

Improve fireball detector recall with stdpixel decontamination#866
alextudorica wants to merge 1 commit into
CroatianMeteorNetwork:prereleasefrom
alextudorica:fireball-stdpixel-decontamination

Conversation

@alextudorica
Copy link
Copy Markdown

@alextudorica alextudorica commented May 19, 2026

Summary

The fireball detector misses events when a bright fireball inflates stdpixel
along its trail within the 256-frame FF block. The threshold formula
(avepixel + k1*stdpixel + j1) exceeds 255, gets clipped to 255 (uint8), and
the fireball becomes invisible at the very pixels where it is brightest (87%
of fireballs saturate at maxpixel=255).

The line finder (find3DLines) is not the problem -- it has zero failures
when given a clean point cloud. The bug is entirely upstream in the threshold
stage.

Changes

VideoExtraction.py -- stdpixel decontamination (main fix)

Before thresholdAndSubsample, replace stdpixel at contaminated pixels with
the background median:

  1. Background reference: std_ref = median(stdpixel) where maxpixel < P90
  2. Contamination mask: maxpixel >= P90 AND stdpixel > 3 * std_ref
  3. Replace: stdpixel[contaminated] = max(1, round(std_ref))

Pure numpy preprocessing. Zero-copy fast path when no contamination is present.

Grouping3Dcy.pyx -- threshold clip 254

Clip threshold to 254 (not 255) so near-saturated pixels (maxpixel=255) can
pass. Without this, maxpixel > 255 is always false for uint8.

Grouping3D.py -- FOV-aware velocity filter

Replace the hardcoded if total > 2 in findCoefficients with
fireball_max_ang_vel (deg/s) converted to px/frame via FOV and resolution.
Addresses the existing TODO: "this limit should be read from config file and
calculated for FOV"
. Falls back to the legacy total > 2 when no config is
passed.

.config / ConfigReader.py -- fireball_max_ang_vel

New config parameter (default 60.0 deg/s). Wider than the meteor detector's
ang_vel_max (35 deg/s) because the fireball detector is recall-first.

Benchmark results on 129-event reference dataset

1755 real FF files, 743 stations (639 detected by stock, 104 missed).

FR-positive recall (686 FFs where stock produced an FR file):

Gate Count
DETECTED 679 (99.0%)
velocity 5
min_frames 2
threshold 0
test_points 0
line_finder 0

Zero threshold failures on real data confirms the decontamination works. The 5
velocity failures also fail with the legacy total > 2 filter (not regressions
from the FOV-aware change -- the decontamination changes the point cloud
geometry which shifts line slopes past the velocity cap in both cases).

Missed station rescue (104 stations where stock produced no FR):

Stock With decontamination
Stations rescued 76/104 (73.1%) 77/104 (74.0%)

Overall station-level recall:

Stock This PR
Stations detected 639/743 (86.0%) ~716/743 (96.4%)

The 716 is an upper bound -- the 7 FF-level failures may cause a small number
of station-level losses (floor: 709/743 = 95.4%).

27 stations remain unreachable: 13 genuinely faint (threshold), 10 too brief
(min_frames < 4), 2 velocity, 2 overexposed (white_avg).

@alextudorica alextudorica force-pushed the fireball-stdpixel-decontamination branch from 7ac217a to 2f18ff2 Compare May 21, 2026 12:56
@alextudorica alextudorica changed the base branch from master to prerelease May 21, 2026 12:57
@alextudorica alextudorica force-pushed the fireball-stdpixel-decontamination branch 3 times, most recently from f88aa48 to b714310 Compare May 27, 2026 22:33
@alextudorica alextudorica force-pushed the fireball-stdpixel-decontamination branch from b714310 to 0c788ea Compare May 29, 2026 15:11
A bright fireball inflates stdpixel along its trail within the 256-frame
FF block, pushing the threshold (avepixel + k1*stdpixel + j1) above 255.
The uint8 clip kills near-saturated trail pixels (250-254).

Fix: before thresholding, replace stdpixel at contaminated pixels
(maxpixel >= P90 AND stdpixel > 3x background median) with the
background median. Pure numpy, no Cython changes needed.

Also: clip threshold to 254 (not 255) so near-saturated pixels can pass,
and replace the hard-coded 2 px/frame fireball velocity cap with a
configurable angular velocity (fireball_max_ang_vel, default 60 deg/s)
converted to a per-sensor px/frame limit at runtime using fps and the
platepar-derived deg/pixel scale.

Tested on 129-event reference dataset (1755 FF files, 743 stations):
FR-positive recall 99.0% (679/686), missed station rescue 74.0%
(77/104), overall station recall 86.0% -> 96.4% (716/743). Zero
threshold failures on real data. Line finder has zero failures in all
686 tests -- the problem was entirely upstream in the threshold stage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant