Improve fireball detector recall with stdpixel decontamination#866
Open
alextudorica wants to merge 1 commit into
Open
Improve fireball detector recall with stdpixel decontamination#866alextudorica wants to merge 1 commit into
alextudorica wants to merge 1 commit into
Conversation
7ac217a to
2f18ff2
Compare
f88aa48 to
b714310
Compare
b714310 to
0c788ea
Compare
A bright fireball inflates stdpixel along its trail within the 256-frame FF block, pushing the threshold (avepixel + k1*stdpixel + j1) above 255. The uint8 clip kills near-saturated trail pixels (250-254). Fix: before thresholding, replace stdpixel at contaminated pixels (maxpixel >= P90 AND stdpixel > 3x background median) with the background median. Pure numpy, no Cython changes needed. Also: clip threshold to 254 (not 255) so near-saturated pixels can pass, and replace the hard-coded 2 px/frame fireball velocity cap with a configurable angular velocity (fireball_max_ang_vel, default 60 deg/s) converted to a per-sensor px/frame limit at runtime using fps and the platepar-derived deg/pixel scale. Tested on 129-event reference dataset (1755 FF files, 743 stations): FR-positive recall 99.0% (679/686), missed station rescue 74.0% (77/104), overall station recall 86.0% -> 96.4% (716/743). Zero threshold failures on real data. Line finder has zero failures in all 686 tests -- the problem was entirely upstream in the threshold stage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The fireball detector misses events when a bright fireball inflates
stdpixelalong its trail within the 256-frame FF block. The threshold formula
(
avepixel + k1*stdpixel + j1) exceeds 255, gets clipped to 255 (uint8), andthe fireball becomes invisible at the very pixels where it is brightest (87%
of fireballs saturate at
maxpixel=255).The line finder (
find3DLines) is not the problem -- it has zero failureswhen given a clean point cloud. The bug is entirely upstream in the threshold
stage.
Changes
VideoExtraction.py-- stdpixel decontamination (main fix)Before
thresholdAndSubsample, replacestdpixelat contaminated pixels withthe background median:
std_ref = median(stdpixel)wheremaxpixel < P90maxpixel >= P90ANDstdpixel > 3 * std_refstdpixel[contaminated] = max(1, round(std_ref))Pure numpy preprocessing. Zero-copy fast path when no contamination is present.
Grouping3Dcy.pyx-- threshold clip 254Clip threshold to 254 (not 255) so near-saturated pixels (
maxpixel=255) canpass. Without this,
maxpixel > 255is always false for uint8.Grouping3D.py-- FOV-aware velocity filterReplace the hardcoded
if total > 2infindCoefficientswithfireball_max_ang_vel(deg/s) converted to px/frame via FOV and resolution.Addresses the existing TODO: "this limit should be read from config file and
calculated for FOV". Falls back to the legacy
total > 2when no config ispassed.
.config/ConfigReader.py--fireball_max_ang_velNew config parameter (default 60.0 deg/s). Wider than the meteor detector's
ang_vel_max(35 deg/s) because the fireball detector is recall-first.Benchmark results on 129-event reference dataset
1755 real FF files, 743 stations (639 detected by stock, 104 missed).
FR-positive recall (686 FFs where stock produced an FR file):
Zero threshold failures on real data confirms the decontamination works. The 5
velocity failures also fail with the legacy
total > 2filter (not regressionsfrom the FOV-aware change -- the decontamination changes the point cloud
geometry which shifts line slopes past the velocity cap in both cases).
Missed station rescue (104 stations where stock produced no FR):
Overall station-level recall:
The 716 is an upper bound -- the 7 FF-level failures may cause a small number
of station-level losses (floor: 709/743 = 95.4%).
27 stations remain unreachable: 13 genuinely faint (threshold), 10 too brief
(
min_frames < 4), 2 velocity, 2 overexposed (white_avg).