diff --git a/.github/workflows/build-candidate.yml b/.github/workflows/build-candidate.yml new file mode 100644 index 00000000..c092a257 --- /dev/null +++ b/.github/workflows/build-candidate.yml @@ -0,0 +1,105 @@ +# DRAFT — promotion pipeline skeleton (see docs/promotion-pipeline.md). +# Builds ONE environment-neutral image from the tip of main, tagged +# candidate- and sha-. The same image is later promoted to +# production unchanged; nothing environment-specific may be baked in here. +name: Build Candidate + +on: + workflow_call: + inputs: + workflow-ref: + type: string + required: false + default: main + description: Branch, tag or commit used when calling the workflow. + target: + type: string + required: true + description: Runtime target (ecs, lambda, cloudrun). Selects registry + build identity. + outputs: + project-name: + description: Project name + value: ${{ jobs.build.outputs.project-name }} + candidate: + description: Candidate tag (candidate-) + value: ${{ jobs.build.outputs.candidate }} + build-number: + description: Build number + value: ${{ jobs.build.outputs.build-number }} + +jobs: + build: + name: Build candidate image + runs-on: ubuntu-latest + timeout-minutes: 30 + permissions: + id-token: write + contents: read + outputs: + project-name: ${{ steps.env.outputs.project-name }} + candidate: candidate-${{ steps.build-number.outputs.build-number }} + build-number: ${{ steps.build-number.outputs.build-number }} + steps: + - name: Checkout Cru Actions & Workflows + uses: actions/checkout@v7 + with: + repository: CruGlobal/.github + ref: ${{ inputs.workflow-ref }} + path: cru-github-actions + + # TODO(draft): environment is passed ONLY to resolve the build role and + # secrets namespace (candidates are prod-bound). It must never influence + # image contents. Open question 1: introduce a `shared` BUILD namespace. + - name: Setup Build environment + uses: ./cru-github-actions/actions/setup-env + id: env + with: + environment: production + role-suffix: ${{ inputs.target == 'ecs' && 'TaskRole' || 'GitHubRole' }} + + - name: Configure AWS credentials + uses: aws-actions/configure-aws-credentials@v6 + with: + aws-region: us-east-1 + role-to-assume: ${{ steps.env.outputs.role-arn }} + + - name: Generate Build Number + uses: ./cru-github-actions/actions/build-number + id: build-number + + # TODO(draft): GCP auth + Artifact Registry login when target == cloudrun + # (mirrors build-cloudrun.yml: google-github-actions/auth@v3 via + # vars.GCP_WORKLOAD_IDENTITY_PROVIDER + vars.GCP_SERVICE_ACCOUNT). + - name: Login to Amazon ECR + if: inputs.target != 'cloudrun' + uses: aws-actions/amazon-ecr-login@v2 + id: ecr + with: + mask-password: true + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v4 + + - name: Export BUILD secrets + uses: ./cru-github-actions/actions/secrets + with: + type: BUILD + + - name: Checkout project + uses: actions/checkout@v7 + with: + ref: main + path: project + + # Env-neutral: no ENVIRONMENT build arg, no env-specific tags. The image + # gets candidate- and sha- and is pushed once. + - name: Build & push candidate + working-directory: project + run: ./build.sh + env: + DOCKER_ARGS: >- + --tag ${{ steps.ecr.outputs.registry }}/${{ steps.env.outputs.project-name }}:candidate-${{ steps.build-number.outputs.build-number }} + --tag ${{ steps.ecr.outputs.registry }}/${{ steps.env.outputs.project-name }}:sha-${{ github.sha }} + --push + + - run: echo "::notice title=Candidate Built::${{ steps.env.outputs.project-name }} candidate-${{ steps.build-number.outputs.build-number }} (sha-${{ github.sha }})" diff --git a/.github/workflows/promote.yml b/.github/workflows/promote.yml new file mode 100644 index 00000000..be033c76 --- /dev/null +++ b/.github/workflows/promote.yml @@ -0,0 +1,108 @@ +# DRAFT — promotion pipeline skeleton (see docs/promotion-pipeline.md). +# Promotes the EXACT image currently running on stage to production: +# resolve the stage digest, deploy that digest to prod (no rebuild), +# then tag it release- as the rollback point. +# +# Intended caller: cru-deploy (workflow_dispatch wrapper). The `production` +# GitHub Environment there provides the human gate (required reviewers); +# the concurrency group below is the shared production lock. +name: Promote to Production + +on: + workflow_call: + inputs: + workflow-ref: + type: string + required: false + default: main + project-name: + description: Project Name + required: true + type: string + deploy-type: + description: Runtime target (ecs, lambda, cloudrun) + required: true + type: string + project: + description: GCP Project (cloudrun only) + required: false + type: string + secrets: + datadog-api-key: + description: DataDog API Key + required: true + +jobs: + promote: + name: Promote ${{ inputs.project-name }} stage image to production + runs-on: ubuntu-latest + environment: production + # The single production lock: a promote, a rollback, and a hotfix for the + # same project can never overlap. Never cancel an in-flight production action. + concurrency: + group: production-${{ inputs.project-name }} + cancel-in-progress: false + permissions: + id-token: write + contents: write + env: + DD_API_KEY: ${{ secrets.datadog-api-key }} + steps: + - name: Checkout Cru Actions & Workflows + uses: actions/checkout@v7 + with: + repository: CruGlobal/.github + ref: ${{ inputs.workflow-ref }} + path: cru-github-actions + + - name: Setup Deploy environment + uses: ./cru-github-actions/actions/setup-env + id: env + with: + project-name: ${{ inputs.project-name }} + environment: production + + - name: Record deploy start + run: echo "DEPLOY_STARTED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$GITHUB_ENV" + + # TODO(draft): per deploy-type, query what is ACTUALLY running on stage + # and resolve it to an immutable digest: + # ecs: aws ecs describe-services / describe-task-definition -> container image + # lambda: aws lambda get-function -> Code.ResolvedImageUri (already a digest) + # cloudrun: gcloud run services describe -> spec image + status digest + # Refusing to promote by tag makes this race-free: what was reviewed is + # what ships, even if a newer candidate landed on stage mid-review. + - name: Resolve stage image digest + id: stage + run: | + echo "TODO(draft): resolve digest for ${{ inputs.deploy-type }}" + echo "digest=sha256:TODO" >> "$GITHUB_OUTPUT" + + # TODO(draft): requires the deploy actions to accept an explicit `image` + # input (docs/promotion-pipeline.md, required change 1). Auth per target: + # GitHubDeployECS / GitHubDeployLambda roles, cru-deploy@ SA. + - name: Deploy stage digest to production + run: echo "TODO(draft): deploy ${{ steps.stage.outputs.digest }} via deploy-${{ inputs.deploy-type }} action" + + # TODO(draft): allocate release number (monotonic per project — reuse the + # DynamoDB counter pattern) and add release- to the promoted digest. + # Needs registry tag-write on the deploy identities OR a release ledger + # instead (docs/promotion-pipeline.md, open question 3). + - name: Tag release + id: release + run: echo "TODO(draft): tag ${{ steps.stage.outputs.digest }} as release-" + + # Pinned major + continue-on-error: telemetry must never fail a + # production action (see the deployment-mark incident, PR #425). + - name: Record Deployment in Datadog + continue-on-error: true + run: >- + npx @datadog/datadog-ci@5 dora deployment + --service "$PROJECT_NAME" + --env production + --version "${{ steps.stage.outputs.digest }}" + --started-at "$DEPLOY_STARTED_AT" + --finished-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + --skip-git + + - run: echo "::notice title=Promoted::$PROJECT_NAME stage image promoted to production" diff --git a/.github/workflows/rollback.yml b/.github/workflows/rollback.yml new file mode 100644 index 00000000..0a5c45b7 --- /dev/null +++ b/.github/workflows/rollback.yml @@ -0,0 +1,92 @@ +# DRAFT — promotion pipeline skeleton (see docs/promotion-pipeline.md). +# Redeploys a previous release's image to production. No rebuild, ever. +# Shares the production lock and human gate with promote.yml. This same +# workflow is the execution target for severity-gated AUTO-rollback later — +# automation would invoke it only for releases stamped rollback-safe. +name: Rollback Production + +on: + workflow_call: + inputs: + workflow-ref: + type: string + required: false + default: main + project-name: + description: Project Name + required: true + type: string + deploy-type: + description: Runtime target (ecs, lambda, cloudrun) + required: true + type: string + release: + description: Release tag to redeploy (defaults to the previous release) + required: false + type: string + project: + description: GCP Project (cloudrun only) + required: false + type: string + secrets: + datadog-api-key: + description: DataDog API Key + required: true + +jobs: + rollback: + name: Rollback ${{ inputs.project-name }} to ${{ inputs.release || 'previous release' }} + runs-on: ubuntu-latest + environment: production + concurrency: + group: production-${{ inputs.project-name }} + cancel-in-progress: false + permissions: + id-token: write + contents: write + env: + DD_API_KEY: ${{ secrets.datadog-api-key }} + steps: + - name: Checkout Cru Actions & Workflows + uses: actions/checkout@v7 + with: + repository: CruGlobal/.github + ref: ${{ inputs.workflow-ref }} + path: cru-github-actions + + - name: Setup Deploy environment + uses: ./cru-github-actions/actions/setup-env + id: env + with: + project-name: ${{ inputs.project-name }} + environment: production + + - name: Record deploy start + run: echo "DEPLOY_STARTED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$GITHUB_ENV" + + # TODO(draft): resolve the target release — the given release tag, or the + # release immediately before the one currently deployed — to a digest. + - name: Resolve release digest + id: target + run: | + echo "TODO(draft): resolve ${{ inputs.release || 'previous release' }} for ${{ inputs.deploy-type }}" + echo "digest=sha256:TODO" >> "$GITHUB_OUTPUT" + + - name: Deploy release digest to production + run: echo "TODO(draft): deploy ${{ steps.target.outputs.digest }} via deploy-${{ inputs.deploy-type }} action" + + # dora deployment has no --is-rollback; tag it instead. Pinned major + + # continue-on-error per the deployment-mark incident (PR #425). + - name: Record Rollback in Datadog + continue-on-error: true + run: >- + npx @datadog/datadog-ci@5 dora deployment + --service "$PROJECT_NAME" + --env production + --version "${{ steps.target.outputs.digest }}" + --started-at "$DEPLOY_STARTED_AT" + --finished-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + --skip-git + --custom-tags "rollback:true" + + - run: echo "::notice title=Rolled Back::$PROJECT_NAME production rolled back to ${{ inputs.release || 'previous release' }}" diff --git a/docs/promotion-pipeline.md b/docs/promotion-pipeline.md new file mode 100644 index 00000000..75949344 --- /dev/null +++ b/docs/promotion-pipeline.md @@ -0,0 +1,120 @@ +# Promotion Pipeline — draft design + +**Status: DRAFT.** Implementation sketch for the "Flight" half of the Agentic +Development Lifecycle RFC (build once, promote the artifact). Opened as a draft +PR to make the adoption discussion concrete — not for merge as-is. + +**Scope.** Delivery machinery only: candidate builds, stage deploys, promotion, +release tags, rollback, and the production lock. Merge-gate / review-policy +changes are explicitly out of scope — this pipeline works identically whether a +repo requires human PR review or not, which is what lets it ship first. + +## Image flow + +``` +tip of main + │ candidate build (env-neutral) + ▼ +candidate- (+ sha-) + │ deploy to stage (scheduled daily / manual) + ▼ +[ Stage ] ── human review / UAT ──► Promote (human gate, cru-deploy) + │ same digest, no rebuild + ▼ + [ Production ] + │ + ▼ + release- tag ◄── rollback target +``` + +## Naming + +| Tag | Applied | Meaning | +|---|---|---| +| `candidate-` | at build | Environment-neutral image from tip of `main`. `` continues to come from the DynamoDB `ECSBuildNumbers` counter — monotonic per project. | +| `sha-` | at build | Traceability back to the exact commit. | +| `release-` | at promote | Added to the exact digest deployed to production. Monotonic per project. `Rollback = redeploy release-`. | + +The current `-` tags retire per-app after migration. + +## Components + +| Piece | Where | Status | +|---|---|---| +| `build-candidate.yml` reusable workflow | this repo | skeleton in this PR | +| Stage deploy | existing `deploy-*.yml` + candidate tag | needs action change (below) | +| `promote.yml` reusable workflow (digest-based) | this repo | skeleton in this PR | +| `rollback.yml` reusable workflow | this repo | skeleton in this PR | +| Production lock | `concurrency: production-` on promote/rollback/hotfix jobs | in the skeletons | +| Human gate | GitHub Environment `production` on **cru-deploy** with required reviewers | config, not code | +| Hotfix (cut from release tag + back-merge PR) | later phase | not in this PR | +| Rollback-safety detection (migration classifier) | later phase; only gates *auto*-rollback | not in this PR | + +## cru-deploy side (wrappers) + +Two new `workflow_dispatch` workflows in cru-deploy, mirroring the existing +pattern (thin wrappers over the reusables here): + +```yaml +# cru-deploy/.github/workflows/promote.yml +on: + workflow_dispatch: + inputs: + project-name: { required: true, type: string } + deploy-type: { required: true, type: choice, options: [ecs, lambda, cloudrun] } + project: { required: false, type: string, description: GCP project (cloudrun only) } +jobs: + promote: + uses: CruGlobal/.github/.github/workflows/promote.yml@v1 + with: { project-name: ..., deploy-type: ..., project: ... } + secrets: { datadog-api-key: ${{ secrets.DD_API_KEY }} } +``` + +`rollback.yml` is identical plus a `release` input (defaults to the previous +release). Deploy/Promote/Rollback all bind the `production` GitHub Environment +so required reviewers apply, and share one concurrency group per project. + +## Required changes to existing code + +1. **Deploy actions accept an explicit image.** `actions/deploy-ecs`, + `deploy-lambda`, `deploy-cloudrun` currently construct + `/:-` internally (`src/ecs-config.js`, + `src/gcp.js`). They need an optional `image` input (full reference, + digest preferred) that bypasses tag construction. Stage deploys pass + `candidate-`; promote/rollback pass the resolved digest. +2. **Terraform IAM for re-tagging.** Adding `release-` to an existing + manifest requires `ecr:PutImage` (AWS) / Artifact Registry tag write (GCP) + on the **deploy** identities, which today are read-only against registries. + Alternative: keep registries immutable and record releases in a ledger + (DynamoDB) instead of tags. → Open question 3. +3. **Lambda env-neutrality.** `build-lambda.yml` currently bakes + `PROJECT_NAME` / `ENVIRONMENT` / `BUILD_NUMBER` as build args. Env-specifics + move to runtime configuration (Terraform `aws/lambda/app` change). + +## App-repo migration (per repo, Phase 2) + +1. Replace `build-deploy-*.yml` with a candidate workflow: `push: [main]` + builds `candidate-`; stage deploy on schedule (or on-push while piloting). +2. Delete the `staging` branch, `.github/merge-bot.yml`, and the "On Staging" + label flow after cutover. +3. Old workflows remain until Phase 4 — reverting a pilot is a file revert. + +## Open questions + +1. **BUILD secrets for env-neutral images.** Candidate builds can't pull + env-scoped BUILD secrets. Draft assumes the `prod` namespace (candidates are + prod-bound); the better end state is a `shared` namespace or moving + env-variance fully to runtime. +2. **Daily stage build scheduling.** Per-app cron in the app repo (simple, + distributed) vs. a central matrix cron in cru-deploy (one place, needs an + app registry — `CruApplicationInfo` could serve). +3. **Release record of truth.** Registry tag, git tag in the app repo, GitHub + Release, or a DynamoDB release ledger. Draft: registry tag + git tag. +4. **Preview/lab.** Tag-driven preview deploys (per RFC comments) — out of + scope here, worth designing alongside Phase 2. + +## Phasing + +This PR is Phase 1 of the adoption plan. Phase 2 puts pilot repos (one per +runtime) on it with human review still required. Gate changes (Phase 3) are a +separate, later decision per repo.