OTP authentication started failing — SocketException: Connection reset — across several unrelated tenants at roughly the same time. Different customers, different traffic patterns, same symptom, same window. The tempting narrative writes itself: one shared root cause, probably network, case closed once you find the one thing.
That narrative was half right, and being half right made it dangerous.
The Correlation That Was Real
An AWS Personal Health Dashboard networking event in one region genuinely did line up with some of the failures. Timestamps close enough, region correct, plausible mechanism. If I'd stopped there, the RCA would have named the AWS event as the cause for all of it, and it would have been wrong for at least one customer.
The Timestamp That Didn't Fit
Checking each tenant's actual failure timestamps against the AWS event window instead of just eyeballing "around the same time" turned up a tenant whose failures started two and a half hours before the AWS event even began. Whatever broke that tenant's OTP flow, it wasn't the regional networking event — the timeline made that physically impossible.
That's the moment a single unifying story stops being safe to write. One tenant needed its own independent root cause, and forcing it under the same umbrella as the others would have meant closing an incident with the wrong explanation attached to it.
The Theory I Ruled Out on Purpose
A VPN tunnel issue was also floated early — it's usually a reasonable first guess for this class of symptom. I checked it directly, and explicitly ruled it out: the relevant tunnel had been running single-tunnel and stable for years, with nothing in its logs or metrics suggesting involvement. Citing it anyway — because it was a plausible-sounding, easy-to-say explanation — would have misattributed blame to infrastructure that had nothing to do with the failure, in customer-facing communication no less.
Ruling out a theory is still work. It doesn't feel as productive as confirming one, but a wrong explanation that sounds confident is worse than an honest "still investigating" — especially once it's been said to a customer.
Why This Is Worth Writing Down
The discipline here isn't clever technical debugging — it's resisting a shortcut. When several unrelated things break at once and there's a plausible shared cause sitting right there, it's genuinely hard to keep asking "does the evidence actually support this for every case, or just some of them?" The instinct to close the loop with one tidy story is strong, especially under incident pressure with people asking for an ETA.
Checking exact timestamps against the exact event window — not "around the same time," but the actual minute-by-minute record — is what separates a defensible RCA from a plausible-sounding guess.
Takeaways
- When multiple systems fail with the same symptom in the same window, check the actual timestamps per system before assuming one shared cause — "close enough" isn't the same as "confirmed."
- A theory that's almost right is more dangerous than one that's obviously wrong, because it's more likely to get written into the RCA unchallenged.
- Explicitly ruling out a plausible cause — and saying so in the writeup — is as valuable as confirming the real one. It stops the same wrong theory from resurfacing next time.