After migrating a multi-tenant application onto a new platform, login looped forever. The identity provider reported success on every attempt. The callback endpoint returned 200, every time. The user never reached the application. Just — back to the login screen, again.
Every visible signal pointed upstream, toward the identity provider or the authentication logic. Every upstream theory I tested turned out to be wrong.
Four Hypotheses, Four Disproofs
I want to walk through all four, including the ones that failed, because the failed ones are most of what made this hard.
1. A database access error. Early logs showed what looked like a failed database read on the session path. Turned out to be a deliberate fail-closed default triggering under a specific, harmless condition — not a real failure, just a defensively-coded branch doing exactly what it was told to do.
2. Tenant resolution failing on the callback route. Plausible — multi-tenant systems break in exactly this way when tenant context doesn't survive a redirect. Disproved directly: a live diagnostic on the callback route showed tenant resolution working correctly, every time.
3. A rewrite rule altering the session cookie name. The platform migration touched routing config, so a mangled cookie name was a reasonable next guess. Disproved by tracing every request in the chain through server access logs end to end — the cookie value was consistent throughout. Nothing was rewriting it.
4. A slow synchronous third-party API call on the login path. This one was real. Measurable — adding whole seconds to every login. I disabled it as a test. Response time dropped substantially. The loop continued, completely unchanged. A genuine, independent performance problem, sitting right next to the actual bug, wearing a coat that made it look responsible. I kept it as its own fix and didn't let it masquerade as the answer.
Stopping the Guessing
Four theories in, all either disproved or (in the fourth case) real-but-irrelevant, I stopped inferring from source code and log lines and went and got direct evidence instead: captured the actual browser session through a full login attempt. About 90 requests, a 2.7MB HAR file.
Across the entire capture, there was not one Set-Cookie response header. Not malformed, not misnamed — simply never sent, on any response, at any point in the flow.
The session itself was being written to the database correctly, the whole time. Nothing was wrong with session logic, tenant resolution, or the identity provider. The browser was simply never being told a session existed. It had nothing to hold onto, so every subsequent request looked unauthenticated, so the app sent it back to login, forever.
Finding the Actual Bug
Zero Set-Cookie headers meant the problem wasn't in what the session logic decided — it was in how the response got generated. That reframed the search entirely: not "why is the session wrong," but "why does this response never carry the header it's supposed to carry."
The callback handler — inside a third-party library, vendored into the project — had been modified at some point to render a custom loading page using direct output (writing straight to the response stream) instead of returning the framework's response object. That bypassed the middleware stage responsible for attaching any queued cookies to the outgoing response. The page rendered. It looked completely normal. It just never carried the one header that mattered.
The Fix, and the Part That Mattered More
The fix itself was one line: return a proper response object wrapping the same markup, instead of writing to the stream directly. That's it.
What mattered more: that fix lived inside a vendored dependency directory — the kind of path a routine install or dependency refresh would silently overwrite and discard, with no error, no warning, just a fix quietly gone and the bug quietly back. I flagged it as the highest-priority follow-up item and specified the concrete options for making it durable (a proper patch file tracked in version control, or upstreaming the fix, rather than an edit sitting in a directory nobody's supposed to hand-edit). A fix that disappears on the next clean build isn't a fix — it's a bug with a delay timer on it.
Why This Is Worth Writing Down
The bug was two layers removed from every visible symptom: inside vendored third-party code, inside a framework mechanism (middleware attaching queued cookies) that's completely invisible right up until the moment it doesn't run. Every log line, every code path I could inspect directly, looked reasonable. The thing that actually broke was a side effect of a side effect.
And hypothesis four is worth sitting with on its own — a real, measurable, fixable problem that had nothing to do with the actual bug. It would have been easy, under pressure to close the ticket, to fix it, watch response times improve, and claim victory. It wasn't the answer. Saying so, and shipping it as its own fix instead of a false resolution, mattered as much as finding the real cause.
Takeaways
- When source-level reasoning keeps producing plausible theories that don't hold up, stop inferring and go get a direct capture of what's actually happening on the wire — a HAR file will tell you things logs won't.
- A real, measurable, fixable problem found along the way is not automatically the problem you're looking for — verify causation by testing the fix in isolation before calling it done.
- A fix inside vendored or generated code isn't durable until it survives the next dependency refresh — find where the fix actually needs to live before closing the ticket.