Incident Analysis
AWS US-East-1 Outage (Oct 19–20, 2025): Explained for Busy Engineers
About this article. This is an analysis based on public information. It is not an AWS document and we had no access to AWS internal systems. The facts and times below come from AWS's own summary, “Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region”. The diagrams are simplified. Sections that are our own opinion are labeled as such. If AWS updates its summary, the original source takes precedence.
It started with a race condition
AWS reports that the event began at 11:48 PM PDT on October 19, 2025 and ended at 2:20 PM PDT on October 20. AWS names the root cause as a latent race condition in the DynamoDB DNS management system. It produced an incorrect, empty DNS record for DynamoDB's regional endpoint in US-EAST-1, and the automation did not repair it. Services that depend on DynamoDB, including parts of AWS itself, could not connect.
From there AWS describes three distinct periods of impact: DynamoDB API errors, EC2 instance-launch failures, and Network Load Balancer connection errors. Below, we put the reported root cause and cascade into simple visuals, then offer a practical checklist. The lesson is not “leave your cloud.” It is “design for dependency failure and standardize how you ship.”
Our view (opinion): incidents like this are harder on teams with bespoke pipelines and one-off deployments. A well-documented, centrally supported delivery path tends to help teams respond consistently when a dependency fails.
What failed: the race condition, simplified (per AWS)
AWS describes two components. The DNS Planner creates DNS plans. The DNS Enactor applies them to Amazon Route 53, and runs redundantly in three Availability Zones. One Enactor was unusually delayed while the Planner kept producing newer plans. A second Enactor applied a newer plan and then ran a clean-up that deletes much older plans. At about the same time, the delayed Enactor applied its much older plan over the newer one. AWS says the Enactor's earlier “is this plan newer?” check was stale by then. The clean-up then deleted that older plan, which removed all IP addresses for the regional endpoint and left the system in a state that stopped later updates. Operators had to intervene manually.
Key timeline (simplified, per AWS; times PDT)
- 11:48 PM, Oct 19: the regional DynamoDB endpoint's DNS record becomes empty; clients using it cannot connect.
- 12:38 AM, Oct 20: engineers identify DynamoDB's DNS state as the source.
- 2:25 AM: DNS information restored; clients recover as cached records expire by 2:40 AM.
- 2:25 AM–10:36 AM: new EC2 instance launches fail while EC2's internal lease management and network configuration recover from the backlog.
- 5:30 AM–2:09 PM: some Network Load Balancers show increased connection errors; AWS disables automatic NLB health-check failover at 9:36 AM and re-enables it at 2:09 PM.
- 2:20 PM: AWS reports the event ended.
Why it hurt so much: cascading effects (per AWS)
Many services depend on DynamoDB. AWS reports that, once the endpoint failed, services that rely on it saw errors. EC2's internal lease checks for physical servers depend on DynamoDB, so after recovery a backlog delayed new instance launches. The delayed network configuration for new instances then led to failing load balancer health checks, which AWS says alternated between failing and healthy and triggered automatic DNS failover that removed capacity.
Reality check (our view)
Even well-run cloud regions can have serious incidents. Your job is not to predict the exact bug; it is to make your platform degrade predictably when a core dependency fails.
What this means for your architecture (our recommendations)
- Assume DNS can return “nothing.” Use caching resolvers with sensible TTLs and positive/negative caching.
- Design health checks to avoid flapping. Use slower ramps, hysteresis, and circuit breakers for dependencies.
- Avoid self-amplification. Rate-limit autoscalers and use retry backoff so retries do not create traffic storms.
- Prefer multi-AZ with tested failover. Region-level isolation is ideal, but expensive; start with multi-AZ done well.
- Gate changes. Admission policies should reject regressions: unsigned images, missing health probes, mis-labeled services.
The paved-road playbook for CTOs & SREs (our opinion)
In our experience, a platform team that supplies a paved road (templates, policy, and one supported way to ship) makes teams more consistent when a dependency fails. This is our opinion, not a finding from the AWS report.
- Centralize DNS patterns. Provide a blessed resolver configuration and a sidecar/agent for service DNS with caching.
- Standardize health checks. Company-wide defaults: timeouts, success thresholds, failure thresholds, dependency probes.
- Ship a canary by default. Argo Rollouts/Flagger with metric gates. If teams do nothing, it’s safe.
- Use GitOps for environments. Diffs are truth; promotions are PRs; rollbacks are one click.
- Automate post-deploy checks. Synthetic probes, trace sampling bursts, and error budget accounting.
- Emit DORA metrics. Lead time, frequency, CFR, MTTR as first-class pipeline outputs.
- Practice failure. Quarterly game days: kill DNS for a service, break health checks, flip regions for an hour.
Checklist: reduce blast radius this week
- Introduce an internal caching resolver (CoreDNS/Unbound) with negative caching and sane TTLs.
- Make health checks state-aware: backoff after failures; require consecutive passes before re-entry.
- Set egress budgets for retries; cap concurrent throttle storms.
- Turn on readiness gates so traffic never hits unready pods after a control-plane wobble.
- Run a DNS blackout drill against staging with synthetic traffic. Document RTO/RPO results.
- Adopt a release note format per deploy: who, what, change link, image digest, rollback steps.
Questions leaders ask (our answers)
“Do we need multi-region now?”
Maybe. Start with multi-AZ done properly; prove failover times with drills. If your RTO/RPO or regulatory profile demands region isolation, move there with clear ownership and costs. Multi-region without operational maturity just multiplies pain.
“Why didn’t autoscaling save us?”
Autoscaling adds capacity. It does not make an unreachable dependency reachable. Build timeouts and backoff into clients.
“Should we decentralize pipelines so teams can react faster?”
It depends on the organization. In our view, a centralized, well-documented pipeline makes mitigations and policy enforcement more consistent and gives one place to add lessons learned.
Post-mortem template (steal this)
Title: Short, neutral description
When: YYYY-MM-DD HH:MM TZ
Where: region / AZ / environment
Symptoms: what users saw (error rates, latency, impact window)
Root Cause: one-line cause + contributing factors
Timeline:
- T0: trigger
- T1: first alarm
- T2: mitigation X
- T3: resolved
Mitigation:
- Short-term: steps taken (feature flags, health checks, DNS caching)
- Long-term: architectural and pipeline changes
Prevention:
- Tests/drills to add
- Policy/rules to encode (admission, CI checks)
- Runbook updates & ownership
What good looks like (our view)
- Every service ships with standard DNS settings, health probes, and circuit breakers.
- Deployment flow produces DORA metrics + release notes on every run.
- Quarterly drills include DNS failure scenarios and flapping health checks.
- Teams build on a paved road, with documented exits for rare cases.
Sources
- Amazon Web Services, “Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region” (all facts and times attributed to AWS above).