Add product and satellites list support and HTML map caching to pipeline#53
Add product and satellites list support and HTML map caching to pipeline#53cmspeed wants to merge 4 commits into
product and satellites list support and HTML map caching to pipeline#53Conversation
ehavazli
left a comment
There was a problem hiding this comment.
Thanks for the update here. I found two blocking issues before this is safe to merge:
-
The new
product: list[str]support is still only partial insidepipeline.py. Inrun_pipeline,folder_name = config.mode if config.mode else config.productcan now become a list, andconfig.output_dir / folder_namewill fail whenmodeis unset. Downstream,generate_products(..., product=config.product)still treatsproductas a scalar and does checks like"DSWX" in product, so the advertised multi-product workflow is not end-to-end safe yet. -
The new cached/cloud-search inputs are not wired through this repo's own entrypoints.
PipelineConfignow hassearch_dirandsatellites, but the CLI still constructsPipelineConfigwithout those fields, and the standalonesearch/downloadcommands still only expose scalarproductinput. That means the new behavior is effectively inaccessible from this repo's public interface unless a caller buildsPipelineConfigmanually.
…on and isolated output folders
|
Code has been updated to address points 1 and 2 above. Also |
|
Manual review on
CodeRabbit raised 0 issues, but the manual review still found the blockers above. |
ehavazli
left a comment
There was a problem hiding this comment.
Review finding:
Critical: src/disasters/pipeline.py line 509 passes products=[np_prod], which wraps the already-built product list in another list. For multi-product downloads this sends a nested list like [['DSWX-HLS_V1', 'RTC-S1_V1']] to next_pass.run_next_pass.
next_pass.run_next_pass expects products as str | list[str] and extends that list into CLI args, so the nested list can produce malformed product arguments.
Suggested fix:
products=np_prodinstead of:
products=[np_prod] if np_prod else None
This PR updates the
PipelineConfigand core execution functions inpipeline.pyto accept lists for theproductandsatellitesparameters. It also introduces a caching mechanism to port pre-generated footprint maps into individual output folders. To support the new unified workflow in thedisaster-alertsdashboard, the pipeline must handle a two-phase architecture:(1)
pipeline.pymust be able to accept lists of products and satellites to pass down torun_next_passso it can generate a single master footprints map.(2) When looping through individual products to mosaic them,
run_pipelinemust be able to detect the master cached folder and pull that unified HTML map into the active output directory without triggering redundant searches or overwriting the map.IMPORTANT NOTE: This PR requires the
-pand-sparameter updates in therun_next_passfunction within thenext_passmodule to function. Therefore, prior to merging this PR,next_passPR #51 should first be merged.