PR #93 — Confirm the reviewed saved plan through the HCP API

01Overview

Overview

Purpose

Fix the Terraform CI plan-and-apply flow, which required a manual Confirm click in the HCP Terraform web UI and left merged changes unapplied.

Components touched

.github/workflows/terraform.yml
tf/modules/prometheus-stack/tests/workflow_contract_check.py
tf/modules/prometheus-stack/tests/workflow_apply_harness.py
tf/modules/prometheus-stack/tests/task3_contract_tests.py
tf/modules/prometheus-stack/tests/fixtures/workflow-contract-failures.json

External systems

HCP Terraform at app.terraform.io, organization ekho-labs, workspace kubernetes-infra, CLI-driven with no VCS connection; GitHub Actions; GitHub Environments.

02The two defects

The two defects

Defect 1

The apply never sent the confirmation

On Terraform CLI 1.10.2 the cloud backend confirms a saved-plan run only when -auto-approve is false. The workflow passed -auto-approve, so the confirmation API call was skipped; Terraform attached to the remote run and waited for someone else to confirm it. Fixed upstream in hashicorp/terraform#36453.

internal/cloud/backend_apply.go, v1.10.2

if !op.AutoApprove && err != errRunApproved {
    b.client.Runs.Apply(stopCtx, r.ID, tfe.RunApplyOptions{})
}

Timeline

00:06:55 job prints "Preparing the remote apply..." and the run URL
no further output for 4 hours 48 minutes
04:54:14 job cancelled by hand
hours later run confirmed in the HCP web UI, then applied

Defect 2

Pull requests created applyable production runs

On the cloud backend, terraform plan -out does not save a plan locally. It promotes the remote run from speculative to a saved-plan run, which is applyable and waits for confirmation. Every ready-for-review pull request created one in the production workspace, despite the job being named Speculative Plan. HCP discards every other outstanding saved plan as soon as one is confirmed, so the open pull requests could never each complete an apply.

Observed HCP run list

Run id	Type	Status
run-bSEqJffA6dBCZ7Dw	saved plan run	Applied
run-P7myx8rSThpfUHij	saved plan run	Discarded
run-WdqK5Y4BUvDzoPx3	saved plan run	Discarded
run-YskJLyh2QvC8xosB	saved plan run	Discarded

All four at the same timestamp — confirming one auto-discarded the other three.

03Flow

Flow: before vs after

Before

failure point

Lane 01Pull request

01

push to PR branch

02

job terraform-plan

03

terraform plan -out=tfplan

04

HCP saved-plan run, applyable, awaiting confirmation

applyable run created from branch code

Lane 02Merge to main

01

push to main

02

job terraform-main-plan

03

terraform plan -out=tfplan

04

HCP saved-plan run + artifact

Lane 03Apply

01

human triggers workflow_dispatch pasting 4 values (source_run_id, commit_sha, tfplan_sha256, plan_manifest_sha256)

02

job terraform-apply

03

terraform apply -auto-approve tfplan

04

attaches to run but never confirms

hangs indefinitely

05

human clicks Confirm in the HCP UI

06

applied

07

every other outstanding saved plan auto-discarded

After

new guarantee

Lane 01Pull request

01

push to PR branch

02

job terraform-plan

03

terraform plan with no -out

04

HCP speculative plan-only run, not applyable

Lane 02Merge to main

01

merge requires a pull request with code-owner approval of its last push

02

push to main

03

job terraform-main-plan

04

terraform plan -detailed-exitcode -out=tfplan

05

saved-plan run, redacted manifest, digests, artifact, and a has_changes output

Lane 03Branch

has_changes == false

01

apply skipped, job summary records that nothing is required

has_changes == true

01

job terraform-apply via needs

02

nine-assertion gate

03

POST /api/v2/runs/:id/actions/apply requiring HTTP 202

04

poll run status to a deadline

05

applied

no human click, no backlog

04The apply gate

The apply gate

1
The downloaded artifact contains exactly tfplan and plan-review-manifest.json, and both SHA-256 digests equal the values the plan job published.
2
The saved plan pointer carries exactly the keys hostname, remote_plan_format, run_id.
3
remote_plan_format is 1.
4
hostname is app.terraform.io.
5
run_id matches ^run-[A-Za-z0-9]{16}$ and equals the run the plan job published.
6
The run belongs to workspace kubernetes-infra.
7
The run is a saved-plan run, carries changes, is reported confirmable, and the workspace is unlocked.
8
The run status is one of planned_and_saved, cost_estimated, policy_checked.
9
The manifest re-derived from the remote plan equals the reviewed digest, and origin/main still equals this commit.

05Time limits

Time limits

Declared limits

security-check 5 minutes

terraform-static-checks 15 minutes

terraform-plan 30 minutes

terraform-main-plan 30 minutes

terraform-apply 50 minutes

Remote-apply poll deadline 2700 seconds (45 minutes)

060 min

Enforced by the contract test

R1Every job must declare timeout-minutes and none may exceed 60.

R2The remote-apply deadline must expire before the job timeout, so a stall is reported against the HCP run instead of being killed by GitHub.

WhyThe previous default was GitHub's six-hour job limit, which is how a run reached 4 hours 48 minutes.

06Verification

Verification

16apply-gate fixtures

16 apply-gate fixtures execute the workflow's real shell steps with terraform, curl and git replaced by fixture shims. The happy path confirms and reaches applied. These each fail at the expected step:

pointerextra-keymissing-keyrenamed-keywrong-hostnamepointer-run-mismatch

run_staterun-discardedrun-wrong-workspacerun-not-saved-planrun-without-changesworkspace-locked

rebindmanifest-drift

applymain-advancedconfirmation-refusedapply-erroredapply-timeout

44mutation fixtures

44 mutation fixtures each alter one reviewed marker in the workflow and are all rejected by the contract check.

cleanactionlint

actionlint with shellcheck: clean on all new shell. One pre-existing SC2129 style warning remains in the untouched kubeconform step.

6/6contract suite

ruff: clean. Contract suite: 6 of 6 tests pass.

07What to attend

What to attend

Select severities to filter the list. Open a card for the full detail.

Severity

Showing 6 of 6

High

The production environment cannot gate anything.

GitHub returns HTTP 422 for every environment protection rule on this repository: a private repository on the Team plan cannot use required reviewers or even a wait timer. The human gate is therefore the merge itself, which requires a pull request with a code-owner approval of its last push. Accepting this PR means accepting that a code-owner-approved merge applies to production.

Medium

One HCP token is shared between the pull-request plan job and the apply.

A token able to confirm an apply is exposed to branch-controlled workflow code. Splitting it into a plan-only token and an apply token would close this.

Medium

The CLI stays pinned at 1.10.2 to match the workspace.

Confirming through the API makes the 1.10.2 bug irrelevant, but upgrading past hashicorp/terraform#36453 is still worth scheduling.

Low

186 of 626 historical runs in the workspace are in an errored state.

Worth a cleanup pass now that pull requests no longer add runs to it.

Low

A pre-existing SC2129 shellcheck style warning remains in the kubeconform step

untouched by this PR.

Info

Plan artifact retention raised from 1 day to 7 days so it outlives the review window

an expired artifact must force a fresh plan and never a fallback apply.

08Verdict

Verdict

Ready for review

Ready for review. The change is verified by executing the real gate steps against fixtures; the one decision it asks of the reviewer is whether a code-owner-approved merge is an acceptable authorization to apply, given that this repository's plan cannot provide an environment approval.