A banking connector service started returning HTTP 403 Forbidden. The reflex response to a 403 is to go looking for a permissions problem — an expired token, a revoked role, a policy that changed. That reflex would have sent this investigation in the wrong direction, because the response body was telling a different story than the status code.
The Mismatch That Was the Actual Clue
The body of every 403 response read InternalServerError. That's not how a genuine, deliberate 403 usually looks — a real permissions rejection typically comes with a body that says something about permissions, because whatever raised it knew exactly why it was rejecting the request. A 403 wrapped around an internal-error message reads more like something further upstream broke, and whatever caught the failure defaulted to the nearest plausible-looking HTTP status rather than describing what actually happened.
Status codes and error bodies are supposed to agree. When they don't, the disagreement itself is usually more informative than either one taken alone.
Following the Real Thread
Digging into the cluster's secret-sync layer found it: an ExternalSecret resource responsible for syncing a keystore secret from the secret store had failed — Secret does not exist. The connector pods calling the downstream banking API had no valid credentials or certificate to authenticate with at all. Not expired, not revoked — simply never delivered.
Every pod was affected at once, which was itself a useful signal early in the investigation: a single bad instance points at something instance-local (disk, local cache, a bad deploy on one replica), but every replica failing identically at the same time points at something shared — a ConfigMap, a Secret, a sync mechanism — not the application code itself.
A Second, Unrelated Bug Along the Way
While going through the connector's logs during this investigation, I caught something else that had nothing to do with the secret sync: a data-transformation expression computing elapsed request time was throwing null-pointer-style errors. The reason was almost funny in hindsight — it ran inside an error-handling path, computing timing off values that only get set during the normal success path. The error handler itself had a bug that only surfaced once something else had already gone wrong, meaning it had presumably been silently broken for a while, invisible until this incident forced that code path to actually run.
I logged and fixed it separately rather than folding it into the secret-sync writeup — it was real, but conflating two unrelated root causes into one RCA makes both harder to reason about later.
Closing the Loop With the Banking Partner
The downstream banking partner, on their side, asked for the exact error payload and source IP so they could investigate what their systems had seen. Building that evidence package properly mattered: the response body captured verbatim, the egress NAT IP — not the pod's internal cluster IP, which their systems would never actually observe — and the relevant transaction IDs for them to trace on their end. Handing a partner the wrong IP (an internal one they can't cross-reference against anything in their own logs) wastes a round trip that a banking partner's incident process usually can't afford to waste.
Takeaways
- When an HTTP status code and its response body disagree, the disagreement is a clue — don't just read the status code and stop there.
- Identical failures across every replica at the same moment point at shared state (secrets, config, sync mechanisms), not application logic — that should shift where you look first.
- Error-handling code paths can have their own bugs that stay invisible until the exact failure they're supposed to handle actually occurs — worth testing deliberately, not just trusting they work because they're rarely exercised.
- When handing evidence to an external partner, give them data that's meaningful from their vantage point (egress IP, not internal IP) — not just whatever your own systems logged.