Pluvio#1191
Draft
burlingamet wants to merge 39 commits into
Draft
Conversation
…sor type and avro schema
Merge branch 'master' into pluvio # Conflicts: # pipe/pluvio/pluvio_calibration_loader.yaml
Merge branch 'master' of github.com:NEONScience/NEON-IS-data-processing into pluvio # Conflicts: # pipe/pluvio/pluvio_cron_daily_and_date_control.yaml
…e a 0 when all 30 minute points are NA
…, delete unused pipe
…at NA when all data is NA
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Pluvio precipitation processing for precipWeighingv2 by switching to Pluvio-specific QA/QC + stats containers, adding a publication (group/format/package/egress) pipeline chain, and extending the Pluvio R wrappers with new unit tests and minor logic changes.
Changes:
- Replace the legacy
precipWeighingv2_stats_group_and_computepipeline (removed) with updated Pluvio QM/stats processing and update downstream consolidation to consume it. - Add monthly publication control and new pub pipelines for
precipWeighingv2(group → format/package → egress/publish). - Update Pluvio calibration loader/assignment plumbing and add new
wrap.precip.pluvio.flags/wrap.precip.pluvio.statsunit tests + wrapper logic tweaks.
Reviewed changes
Copilot reviewed 17 out of 22 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| pipe/precipWeighingv2/precipWeighingv2_stats_group_and_compute.yaml | Removes the old stats group/compute pipeline (superseded by QM stats path). |
| pipe/precipWeighingv2/precipWeighingv2_qm_stats_group_and_compute.yaml | Updates the Pluvio stats container image tag used for QM/stats computation. |
| pipe/precipWeighingv2/precipWeighingv2_analyze_pad_and_qaqc.yaml | Updates Pluvio flags image tag and removes passing a (now-optional) plausibility schema. |
| pipe/precipWeighingv2/precipWeighingv2_level1_group_consolidate_srf.yaml | Updates group prefix and points consolidation at the QM stats pipeline output. |
| pipe/precipWeighingv2/precipWeighingv2_cron_monthly_and_pub_control.yaml | Adds monthly cron/pub control pipeline to drive publication timing. |
| pipe/precipWeighingv2/precipWeighingv2_pub_group.yaml | Adds publication grouping pipeline for monthly publication staging. |
| pipe/precipWeighingv2/precipWeighingv2_pub_format_and_package.yaml | Adds formatting + packaging workflow prior to egress/publish. |
| pipe/precipWeighingv2/precipWeighingv2_pub_egress_and_publish.yaml | Adds egress-to-bucket and publish/sync workflow (includes MDP logic). |
| pipe/precipWeighingv2/pipe_list_precipWeighingv2.txt | Updates pipeline list to include new pub chain and consolidation. |
| pipe/pluvio/pluvio_cron_daily_and_date_control.yaml | Adjusts start/end dates and disables cron via @never (consistent with repo patterns). |
| pipe/pluvio/pluvio_calibration_loader.yaml | Simplifies loader command structure and updates env vars/output path. |
| pipe/pluvio/pluvio_calibration_assignment.yaml | Switches assignment input repo from test files to the real calibration loader output. |
| flow/flow.precip.pluvio.stats/wrap.precip.pluvio.stats.R | Adjusts NA aggregation logic and updates finalQF composition behavior. |
| flow/flow.precip.pluvio.flags/wrap.precip.pluvio.flags.R | Changes heater flagging logic to use inlet temperature and tweaks schema handling on write. |
| flow/tests/testthat/test-wrap-precip-pluvio-stats.R | Adds unit tests for Pluvio stats wrapper output and behavior. |
| flow/tests/testthat/test-wrap-precip-pluvio-flags.R | Adds unit tests for Pluvio flags wrapper output and behavior. |
| flow/tests/testthat/pfs/precipPluvioStats/.../pluvio_*_uncertaintyCoef.json | Adds test fixture uncertainty coefficient JSON. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.