Misleading Healthy Metrics Masked AWS Ingress Outage

During a live migration cutover, a production service experienced a 14-hour outage despite monitoring dashboards showing all health signals as green. The root cause was an ingress design using a public NLB (with fixed Elastic IPs) forwarding to an internal ALB; the NLB’s automated HTTP health probes could not inject a Host header, so hardened ALB listener rules returned HTTP 400 and new ALB nodes failed health checks and never entered service. CloudWatch 1-minute Average rollups smoothed sub-minute target churn, hiding the problem. The team fixed it by adding a priority ALB listener rule that matches the NLB source IPs and returns a 200 fixed response for probes (implemented via Terraform). An eight-month postmortem revealed the upstream static-IP requirement was obsolete, exposing an organizational assumption that drove unnecessary complexity. Lessons include separating probe paths from app security, alerting on sub-minute churn, and using synthetic end-to-end checks.