Repository navigation
chore(iac/aws): fck-nat default route blackholes egress after ASG instance replacement #716
Description
Summary
The AWS private IPv4 default route is pinned to the fck-nat ENI resolved at plan time. When the ASG replaces the instance, the route points at a dead ENI and egress is blackholed until the next terraform apply.
Where
At commit b50ca38: terraform/modules/networking/aws/main.tf:326-357,483-489.
Failure scenario
The fck-nat instance is replaced by the ASG (health check, spot reclaim, AZ event). The new instance gets a new ENI, the route table still references the old one, and private subnets lose outbound access (Lambda, Fargate, RDS-adjacent calls to AWS APIs). Found by source reading; not reproduced live.
Suggested fix
Use a persistent ENI that the instance attaches at boot, or use a NAT Gateway in prod. Add VPC endpoints for the AWS services the workloads call.
Severity
High: an ordinary instance replacement causes a full egress outage that needs manual intervention.
Reviewer ID: OPS-05 (ops H5)
Found by the 2026-10-08 review fleet.
Activity
IAmJSD commentedon Oct 19, 2026
Claiming: working on this
IAmJSD commentedon Oct 27, 2026
Status update (2026-10-27, checked against main at 7b86e38): Waiting on a held PR.
Main still routes private egress at data.aws_network_interfaces.fck_nat[0].ids[0] (terraform/modules/networking/aws/main.tf:488), so an ASG replacement still blackholes egress. Held draft PR #1090 (D7) moves the routes to a persistent ENI; it needs a manual ci-cd-permissions apply before merge.
Left / blocked on: Held open PR #1090 (D7).