A credit union tenant's members suddenly couldn't retrieve their account balances. The on-call alert pointed at something familiar and usually boring: a VPN tunnel flap. The kind of blip that shows up, self-heals in a minute or two, and gets closed out with a one-line note. That's what everyone expected here too.
It didn't self-heal. The outage ran for hours, and closing the ticket as "network blip, case closed" would have been the easy, wrong answer.
The Obvious Cause Wasn't Wrong — It Was Incomplete
The VPN tunnel really did flap. That part of the story was true. But a tunnel flap lasting a few seconds doesn't produce a multi-hour customer-facing outage on its own — not unless something downstream turns a transient blip into a standing failure. So instead of stopping at the network layer, I kept going one layer up the stack.
That's where it got interesting. The API layer sitting behind the tunnel — call it sys-api — was running with the HTTP client's default connection pool settings. Nobody had ever sized it for the tenant's real traffic; it had simply never mattered before, because nothing had ever stressed it hard enough to notice. At the same time, that service was already CPU-starved, closer to its ceiling than anyone had flagged.
Put those two facts next to the VPN flap and the picture changes completely:
- The tunnel flaps for a few seconds. Connections in flight fail or stall.
- Retries and reconnects pile up against a connection pool that was never sized for a burst like this.
- The CPU-starved API service can't clear the backlog fast enough to keep up, so the queue keeps growing instead of draining.
- What should have been a few seconds of noise becomes hours of degraded service, because the system had no slack left anywhere to absorb it.
Three Factors, Not One
This is the part worth sitting with: none of these three factors was, by itself, the incident. A brief VPN flap happens periodically and is usually invisible. A slightly undersized connection pool is invisible too, right up until something else fills it faster than expected. A CPU-starved service that's "fine" under normal load is fine — until it isn't asked to absorb a sudden backlog.
It took all three showing up in the same window to turn a non-event into a multi-hour outage. If I'd stopped at "VPN flapped, and then it recovered" — which is exactly what the first alert suggested — the connection pool and the CPU headroom problem would still be sitting there today, waiting for the next transient network hiccup to rediscover them.
What Actually Shipped
Two concrete pull requests came out of the RCA:
- Connection pool tuning on the HTTP client used by
sys-api, sized against realistic peak concurrency instead of the framework default. - Resource allocation changes to give the service actual CPU headroom instead of running close to its ceiling under normal load.
And the RCA itself was written to walk the entire causal chain — network, then connection handling, then compute — rather than naming a single root cause and moving on. That distinction mattered more than the specific fixes. A root cause analysis that stops at the first plausible explanation teaches the team nothing about the other two problems still sitting in production.
The Takeaway
Default settings are load-bearing until the day they aren't. A connection pool sized by a framework default, a service running without headroom — these aren't bugs in the traditional sense. They're latent capacity problems that stay invisible for as long as nothing stresses them. The job of an RCA isn't just to explain what happened; it's to keep asking "and then what?" until you run out of contributing factors, not until you run out of patience.
The next time an incident resolves itself with a plausible one-line explanation, it's worth asking: is that actually the whole story, or just the first domino?