Don't Let the Vendor Write the Incident Timeline

A Sev-1 hit a customer's site-to-site connectivity through an AWS Transit Gateway ↔ Azure Virtual WAN path. Routes disappeared, then came back. AWS's initial read on their side: looked self-resolving, roughly three minutes, no action needed.

That explanation would have closed the ticket. It just wasn't what actually happened.

The Data That Was Already Running

Independent of anything AWS could see, I had a monitoring script quietly snapshotting BGP route counts, tunnel state, and target health every few minutes, all the way through the incident window. It wasn't built for this specific outage — it was just standing infrastructure, running because good monitoring runs whether or not you're currently having a bad day.

When I lined those snapshots up against AWS's stated timeline, two things fell out immediately:

• The BGP routes were already absent before AWS's own stated maintenance window even began.
• Recovery only ever happened after a manual tunnel replacement on our side — never on its own, at any point in the data.

In other words: not self-resolving, and not caused by the maintenance window AWS initially pointed to. Two claims, both testable, both wrong.

Finding the Actual Root Cause

With the "it healed itself" theory off the table, the question became: why were the routes gone at all? The answer turned out to be a single misconfigured line on the Azure side of the tunnel pair.

Tunnel 2 was advertising all four of the customer's network prefixes, but with Tunnel 2's own peer address pointed at Tunnel 1's next-hop instead of its own. That's invalid per RFC 4271 — a BGP NEXT_HOP has to be reachable and correct for the session it's advertised on. Because of the misconfiguration, Tunnel 2 could never actually carry usable routes, even though its session showed as "up." It looked healthy. It wasn't.

# what "looks up" doesn't tell you
Tunnel 1: session UP, NEXT_HOP correct    → routes usable
Tunnel 2: session UP, NEXT_HOP misrouted  → routes advertised, never usable

An AWS engineer, once shown the timeline, confirmed the event as customer-side-initiated rather than an AWS platform fault — and validated the manual-recovery timeline over the original self-resolving narrative.

Why This Is Worth Writing Down

This wasn't about proving anyone wrong for its own sake. It's about what happens when you don't have your own instrumentation: you're stuck accepting whatever the vendor's dashboard says, because you have nothing to compare it against. A cloud provider's status page and internal telemetry are built to answer their operational questions, not necessarily to reconstruct the specific few minutes that mattered to your specific customer.

The monitoring script that made this possible wasn't sophisticated — route counts, tunnel state, target health, on a timer, written to somewhere durable. What made it valuable was that it existed before the incident and kept running through it, so there was a real, timestamped record to check the vendor's story against instead of just a memory of "it felt like it was already broken."

Takeaways

• Independent monitoring earns its keep exactly in the moment a vendor's explanation doesn't match what you're seeing.
• "Looks up" and "actually usable" are different claims for a BGP session — a session can be up with routes that can never work.
• Bring timestamps, not opinions, when a vendor's timeline doesn't match your evidence. It's a much shorter conversation.