Promotion 2026-03-31 prod (#7900)

[achave11-ucsc]

Linked issue: #7900

Notes

• Perform a manual deletion of the dcp57/-it catalogs in prod
• Requires updating the CloudWatch saved query Clamscan status in the respective AWS consoles when merged to lower & stable branches.

Checklist

Author

• PR is assigned to the author
• Status of PR is In progress
• Target branch is prod
• Name of PR branch matches promotions/yyyy-mm-dd-prod
• PR is linked to the promotion issue it resolves
• Status of linked issue is In progress
• PR description links to linked issue
• Title of linked issue matches Promotion yyyy-mm-dd
• PR title starts with title of linked issue followed by prod
• PR title references the linked issue

Author (reindex)

• This PR is labeled reindex:prod _{or the changes introduced by it will not require reindexing of prod}
• This PR is labeled reindex:partial and its description documents the specific reindexing procedure for prod _{or requires a full reindex or is not labeledreindex:prod}

Author (mirror)

• This PR is labeled mirror:prod _{or the changes introduced by it will not require mirroring of prod}
• This PR is labeled mirror:partial and its description documents the specific mirroring procedure for prod _{or requires a full mirroring or is not labeledmirror:prod}

Author (upgrading deployments)

• This PR is labeled upgrade _{or does not require upgrading deployments}
• This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
• This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
• This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (before every review)

• PR branch is up to date (if not, merge prod into PR branch to integrate upstream changes)
• PR is not a draft
• PR is awaiting requested review from system administrator
• Status of PR is Review requested
• PR is assigned to only the system administrator and the author

System administrator (after approval)

• Actually approved the PR
• Labeled PR as no sandbox
• N reviews label is accurate
• Status of PR is Approved
• PR is assigned to only the operator and the author

Operator

• Pushed PR branch to GitHub

Operator (deploy .shared and .gitlab components)

• Ran _select prod.shared && CI_COMMIT_REF_NAME=prod make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select prod.gitlab && python scripts/create_gitlab_snapshot.py --no-restart (see operator manual for details) _{or this PR is not labeled backup:gitlab}
• Ran _select prod.gitlab && CI_COMMIT_REF_NAME=prod make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Checked the items in the next section _{or this PR is labeled deploy:gitlab}
• PR is assigned to only the system administrator and the author _{or this PR is not labeled deploy:gitlab}

System administrator (post-deploy of .gitlab component)

• Background migrations for prod.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• PR is assigned to only the operator and the author

Operator (deploy runner image)

• Ran _select prod.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}

Operator (merge the branch)

• All status checks passed and the PR is mergeable
• The title of the merge commit starts with the title of this PR
• Added PR # reference to merge commit title
• Collected commit title tags in merge commit title _{but excluded any p tags}
• Pushed merge commit to GitHub
• Status of PR is Merged stable

Operator (main build)

• Pushed merge commit to GitLab prod
• Build passes on GitLab prod
• Reviewed build logs for anomalies on GitLab prod
• Ran _select prod.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Deleted PR branch from GitHub
• PR is assigned to only the operator
• Status of linked issue is Stable
• Status of promoted¹ PRs is Merged stable
• Status of promoted¹ issues is Stable

¹ Promoted issues and PRs are referenced in the titles of the commits
that the promotion branch introduces to the stable branch. Prior to the
promotion, the status of promoted issues (PRs) is Lower (Merged lower).
Promoted PRs in status Done do not need to be moved.

Operator (reindex)

• Deindexed all unreferenced catalogs in prod _{or this PR is neither labeled reindex:partial nor reindex:prod}
• Deindexed specific sources in prod _{or this PR is neither labeled reindex:partial nor reindex:prod}
• Indexed specific sources in prod _{or this PR is neither labeled reindex:partial nor reindex:prod}
• Started reindex in prod _{or this PR does not require reindexing prod}
• Checked for, triaged and possibly requeued messages in both fail queues in prod _{or this PR does not require reindexing prod}
• Emptied fail queues in prod _{or this PR does not require reindexing prod}
• Restarted the Data Browser pipeline for the ucsc/hca/prod branch on GitLab in prod _{or this PR does not require reindexing prod}
• Restarted the Data Browser pipeline for the ucsc/lungmap/prod branch on GitLab in prod _{or this PR does not require reindexing prod}
• Restarted deploy_browser job in the GitLab pipeline for this PR in prod _{or this PR does not require reindexing prod}

Operator (mirroring)

• Started mirroring in prod _{or this PR is not labelled mirror:prod}
• Checked for, triaged and possibly requeued messages in mirror fail queue in prod _{or this PR is not labelled mirror:prod}
• Emptied mirror fail queue in prod _{or this PR is not labelled mirror:prod}

Operator

• PR is assigned to only the system administrator

System administrator

• Removed unused image tags from pycharm image on DockerHub _{or this promotion does not alter references to that image}
• Removed unused image tags from bigquery_emulator image on DockerHub _{or this promotion does not alter references to that image}
• PR is assigned to no one

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.93%. Comparing base (4c5b317) to head (7a1f9a5).
⚠️ Report is 12 commits behind head on prod.

Additional details and impacted files

@@           Coverage Diff           @@
##             prod    #7906   +/-   ##
=======================================
  Coverage   84.93%   84.93%           
=======================================
  Files         161      161           
  Lines       23130    23130           
=======================================
  Hits        19645    19645           
  Misses       3485     3485           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coveralls]

coverage: 85.076%. remained the same
when pulling 7a1f9a5 on promotions/2026-03-31-prod
into 4c5b317 on prod.