Update lakeflag for misclassified reaches #19

Activity

1. jameshgrn

   added this to the v17c-lake-type milestone on Jan 26, 2026
2. jameshgrn

   added

   P1-highHigh priorityHigh priority

   type:bugBug fixBug fix

   region:allAll regionsAll regions

   comp:lake-typeComponent: lake-typeComponent: lake-type

   on Jan 26, 2026
3. jameshgrn commented on Jan 28, 2026

   

   jameshgrn

   on Jan 28, 2026

   CollaboratorAuthor

   More actions

   Update: Validation GUI (#75) now supports lakeflag fixes:

   • C001 tab: Convert river→lake for lake sandwiches
   • C004 tab: Fix lakeflag/type mismatches

   Both tabs include satellite imagery, skip reasons, and local persistence.
4. jameshgrn

   modified the milestones: v17c-lake-type, v17c-april-2026 on Feb 3, 2026
5. jameshgrn commented on Feb 6, 2026

   

   jameshgrn

   on Feb 6, 2026

   CollaboratorAuthor

   More actions

   Lake Sandwich Correction Complete (v17c)

   1,252 reaches corrected — river reaches (lakeflag=0) sandwiched between lake reaches, reclassified to lakeflag=1.

   Detection Criteria

   • Width: sandwich width >= 50% of the wider neighboring lake
   • Length: sandwich shorter than at least one neighboring lake reach
   • Both upstream AND downstream neighbors must be lakeflag=1

   Changes Applied

   • lakeflag → 1
   • type → copied from wider lake neighbor (mostly type=3)
   • edit_flag → appended lake_sandwich for traceability

   Regional Breakdown

   Region	Count
   NA	548
   AS	344
   EU	161
   SA	143
   AF	43
   OC	13

   Not Corrected (requires manual review)

   • Narrow sandwiches (width < 50% of lakes): ~732 — high FP rate, many are genuine connecting channels
   • Chain sandwiches (2+ consecutive river reaches between lakes): ~949 pairs — need separate approach
   • Wide + longer than both lakes: ~170 — mixed confidence

   Synced to

   • DuckDB: data/duckdb/sword_v17c.duckdb ✅
   • PostgreSQL: reaches table ✅

   Review files

   • output/facc_detection/lake_sandwich_wide_shorter_one.geojson (the 1,252 corrected)
   • output/facc_detection/lake_sandwich_high_confidence.geojson (956 subset: shorter than both lakes)
6. jameshgrn

   added

   target:2026-03-06Target completion date 2026-03-06Target completion date 2026-03-06

   on Feb 21, 2026
7. jameshgrn

   self-assigned this

   on Feb 21, 2026
8. jameshgrn commented on Mar 13, 2026

   

   jameshgrn

   on Mar 13, 2026

   CollaboratorAuthor

   More actions

   Reviewer Fix Recovery (2026-03-13)

   Discovered and fixed 3 bugs in the reviewer app that caused Caroline's reviewed reaches to reappear after Cloud Run container restarts:

   1. State only read from ephemeral DuckDB — now reads from both DuckDB and GCS FUSE JSON sessions
   2. Non-atomic JSON writes — open("w") could corrupt files on container kill; now uses temp file + fsync + rename
   3. No error handling for corrupted JSON — crashes instead of graceful degradation

   Recovered data

   Source	Fixes	Skips
   GCS JSON sessions (3 files, all intact)	72 active	513
   DuckDB lint_fix_log (local)	142 active	306
   PostgreSQL lint_fix_log	19 active (older sync)	303

   After cross-source deduplication:

   • 25 fixes existed only in GCS (Mar 3–13 work), now synced to DuckDB + PostgreSQL
   • 149 skips existed only in GCS, now synced to both databases
   • No data was lost — the GCS JSON files survived every container restart

   Current totals (DuckDB lint_fix_log)

   • 167 active fixes (mostly C001 lakeflag corrections in NA)
   • 455 skips
   • 622 total rows

   Caroline's manual review progress (C001 NA)

   • 63 active lakeflag fixes (river→lake)
   • 501 reaches reviewed and skipped as correct
   • Work spans Feb 18 – Mar 13

   Fixes applied

   • Commits 1d58732 and 6483c84 on reviewer-edits-problem branch
   • Recovery backup at gs://sword-qc-data/sword/recovery/reviewer_recovery_2026-03-13.json
9. jameshgrn commented on Mar 25, 2026

   

   jameshgrn

   on Mar 25, 2026

   CollaboratorAuthor

   More actions

   Closing — lakeflag corrections complete for v17c

   Work completed:

   • 1,252 lake sandwich reaches corrected (lakeflag 0→1, tagged edit_flag='lake_sandwich')
   • 7,425 HarP lake corrections applied (lakeflag 0→1, tagged edit_flag='harp_lake')
   • 127 manual C001 fixes by Caroline via reviewer app (NA, lakeflag 0→1)
   • 10 manual C004 fixes by Caroline (NA, type reclassifications)
   • Reviewer app persistence bugs fixed (GCS JSON + DuckDB + PostgreSQL sync)
   • All corrections propagated to nodes

   Remaining (acceptable for v17c):

   • ~732 narrow sandwiches (width < 50% of lakes) — high FP rate, left for manual review
   • ~949 chain sandwich pairs — need separate approach
   • type/reach_id reconciliation deferred to v18 (Reconcile type field with lakeflag for HarP/lake-sandwich corrections #208)

   Exports updated:

   NetCDF, GPKG, GeoParquet, PostgreSQL all include corrections.
10. jameshgrn

    closed this as completedon Mar 25, 2026