Repository navigation
Conversation
…logs after a failed health check The Azure Terraform Apply step read its outputs with `terraform output -raw X 2>/dev/null | grep -v '::' || echo ""`, which hid terraform's error and published an empty step output. The failure then surfaced a job later as "Failed to get app URL" or "Terraform outputs missing", far from its cause. The outputs are now read by scripts/read-tf-outputs.sh, which keeps terraform's stderr, still drops the setup-terraform wrapper's `::` lines, requires exactly one non-empty value line per output and writes nothing to GITHUB_OUTPUT unless every output was read. The "Check Container App logs" step had no condition, so it was skipped exactly when the readiness or smoke gate failed. It now runs with `always()` once the Azure login succeeded, and warns instead of failing when there is no app to read from; an az lookup error still fails it. scripts/test-read-tf-outputs.sh tests the script with a stub terraform and executes the workflow's own Apply and log-step run blocks with stub terraform/az; it runs in CI next to the smoke-gate self-test. test-wait-for-healthy.sh gains the empty-body and non-JSON response cases LeanerCloud#217 asked for. Closes LeanerCloud#438 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
Follow-ups from the review of #437, each re-checked against current main (
a14dc8f2):deploy-azure.yml,test-deployment/ "Check Container App logs" had noif:. So it was skipped exactly when "Test readiness endpoint" or "Run smoke tests" failed, which is when the logs are needed.terraform output -raw X 2>/dev/null | grep -v '::' || echo "". That threw away terraform's error and published an empty step output. The failure then surfaced a job later as "Failed to get app URL" or "Terraform outputs missing", far from the cause.scripts/wait-for-healthy.sh, andscripts/test-wait-for-healthy.sh(run in CI) already covers healthy, degraded/503, connection errors and 301. The empty-body and non-JSON shapes that fix(ci): fail Azure deployment verification on degraded health and log lookup errors #217 named were still missing.--max-time 60nit (already fixed). fix(server/azure): exclude uninitialized replicas from readiness #488 removed the warm-up request. The gate's only curl uses--max-time 10.Fix
scripts/read-tf-outputs.sh KEY=OUTPUT_NAME..., called from the Apply step in place of the three inline lines. It:terraform output -raw NAMEand leaves terraform's stderr in the log;::error::(exit 1) if terraform fails, or if the value isn't exactly one non-empty line after dropping lines that contain::(the setup-terraform wrapper's::debug::noise, the same filter as 3d0d9de);$GITHUB_OUTPUTonly after every output has been read, so a failure never leaves a half-written set.if: always() && steps.azure-login.outcome == 'success', and the Azure Login step in that job getsid: azure-login. It runs after a failed gate but not when there is no Azure session to query.::warning::and exit 0. With item 2 fixed, build-and-deploy fails at the source when it can't read them, so failing again here would only bury the real error.azlookup error still fails the step, as fix(ci): fail deploy verification on degraded health and Azure log lookup errors #437 intended for fix(ci): fail Azure deployment verification on degraded health and log lookup errors #217.scripts/test-read-tf-outputs.sh, added to CI'sazure-kv-access-policyjob next to the existing smoke-gate self-test. It:terraform;run:blocks and executes them with stubterraform/azon PATH (the same extraction helpers astest-migration-down-gate.sh);always()condition and theazure-loginid it refers to;terraform outputcall indeploy-azure.ymlstill pipes to2>/dev/nullor|| echo.scripts/test-wait-for-healthy.shgains four cases: 503 with an empty body, 502 with an HTML (non-JSON) body, that 502 resetting a streak, and 200 with an empty body counting as ready, because the status code is the contract.Closes #438
Deploy pause: when this matters, and why merging is safe
Azure deploys have been paused since 2026-10-20 (deploy-azure/gcp/all workflows disabled). Until they are re-enabled this changes nothing at runtime; it takes effect on the first Azure deploy after the pause lifts. Merging is safe while paused: the only change that runs on every PR is one new offline step in
ci.yml, a shell test with stub binaries and no cloud or network access.There is one behavior change to know about for that first deploy. If
terraform applysucceeds but an output can't be read, the Apply step now fails with terraform's error, instead of passing and failing later intest-deployment. That is the intent of this issue.Verification
All of this is offline evidence from stubs and fixtures, run on the branch head. No workflow was dispatched and no terraform or az command reached a real backend.
Fail before / pass after. The workflow half of the new test was run against
origin/main'sdeploy-azure.yml(WORKFLOW=<main copy> bash scripts/test-read-tf-outputs.sh) and gavepassed=21 failed=5:workflow apply: a missing output did not fail the step loudly (exit 0)workflow apply: published 'resource_group_name='workflow: a terraform output error is still swallowed(lines 286, 290, 293)workflow logs: step has no always() condition ...workflow logs: missing outputs (exit 1 ...)All five pass on this branch.
Mutation probes on
read-tf-outputs.sh. Each probe makes the suite fail:::filter removedMerge check.
git merge-treeagainst the open branches that touchdeploy-azure.ymlorci.yml(#1147, #1222, #1227, #1119, #1142, fix/issue-1233, -177, -59): no conflicts.Combined-tree verification. This branch was tested together with the other queued branches on a local, never-pushed tree:
queue/platform@b56afb2b= origin/maina14dc8f2plus these heads, merged in this order:fix/issue-324andfix/issue-204both edit.github/workflows/ci.ymlnext to theazure-kv-access-policyjob. The tree kept #324's new "Run Function URL reader self-tests" step and #204's updated job comment; whichever PR merges second takes that same resolution.Linux, Go 1.26.9,
AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true; Docker Postgres via testcontainers. The full suite ran oned0c7174, which differs fromb56afb2bonly by one.gitallowedline on fix/issue-324:go build ./...,go vet ./...,go vet -tags integration ./...and the fullgo test -race -short -count=1 ./....--build-tags integration.gocyclo -over 10is silent on the changed non-test files.-tags integration -count=1, 60-minute local timeout):internal/api,internal/purchase,internal/schedulerandinternal/serverall pass..gitallowedline. Re-run atb56afb2b: git-secrets, using the repo'sscripts/setup-git-secrets.shpatterns in a fresh repo with an isolated HOME, found nothing in the changed files, and a planted fake key was flagged.scripts/test-git-secrets-allowlist.sh: 137 PASS, 0 FAIL.Out of scope
deploy-aws-fargate.yml,deploy-aws-lambda.ymlanddeploy-gcp.ymluse the sameterraform output ... 2>/dev/null | grep -v '::' || echo ""pattern. This issue is about Azure only.read-tf-outputs.shtakes genericKEY=OUTPUT_NAMEpairs, so those workflows can adopt it in a follow-up.grep -v '::'wrapper filter is kept as it was. Settingterraform_wrapper: falseon setup-terraform would remove the noise at the source, but it changes plan/apply annotations for the whole job, so it is a separate decision.Labels
A maintainer, please mirror the issue's triage labels onto this PR:
impact/internal,type/chore,triaged,priority/p3,severity/low,urgency/eventually,effort/s.🤖 Generated with Claude Code