A developer reported vague performance issues on a cluster. Nothing specific — things just felt slow, intermittently, in a way that was hard to pin to any one request or service. Dashboards showed nothing down. No alerts firing, no red anywhere.
That combination — "nothing's red, but someone's unhappy" — is one of the least satisfying categories of investigation. There's no stack trace to follow, no obvious starting point. The easy failure mode here is to poke around for ten minutes, find nothing alarming, and close it out as "couldn't reproduce." I dug anyway, and found four distinct, real problems, none of which alone would have paged anyone.
Problem One: Zombie Nodes
Three nodes had been stuck in a NotReady state for as long as 361 days — leftovers from a nodegroup migration that nobody had gone back to clean up. They were silently holding five pods stuck in Terminating forever. I confirmed none of them were still receiving traffic (checked they weren't in any load balancer's endpoint list), so this wasn't actively breaking anything. It was still real waste, and real noise every time someone ran kubectl get nodes and had to mentally filter out the zombies.
Problem Two: An End-of-Life Autoscaler
The cluster autoscaler was running a version actively failing its own version-compatibility checks against the current Kubernetes control plane version. "Limping, not dead" — it was still functioning, but every scaling decision was happening on a code path nobody was validating anymore, on borrowed time until an incompatibility actually bit.
Problem Three: A Phantom Provisioning Loop
This one was actively costing money for nothing. The autoscaler was launching a fresh node for a pending pod roughly every 60-90 minutes — the pod never actually landed on it, and the node got consolidated away five minutes later. Then it happened again. All day. Prime suspect: a topology spread constraint fighting the autoscaler's own placement logic, each one convinced the other was wrong about where the pod should go.
Problem Four: Real Memory Pressure
And here's the one that was almost certainly what the developer was actually feeling: nodes running at 89-93% memory utilization, with several pods exceeding their configured memory requests under real load. That combination produces garbage-collection-driven latency spikes — the kind of "things just feel slow sometimes" symptom that's genuinely hard to describe precisely, because it comes and goes with GC pauses rather than presenting as a clean failure.
The Order Mattered
Finding four problems is one thing. Fixing them in the wrong order would have made things worse. The sequencing I recommended was deliberate:
- Clean up the dead nodes first — zero risk, immediate noise reduction.
- Fix the memory pressure — the thing actually causing the reported pain.
- Upgrade the autoscaler — remove the compatibility time bomb.
- Only then consider Pod Disruption Budgets.
That last one is worth explaining. PDBs were a real, legitimate gap in this cluster's configuration. But they weren't the root cause of anything reported, and setting one too strictly — say, maxUnavailable: 0 — before cleaning up the zombie nodes and stale autoscaler would have actively blocked node consolidation and made the existing mess harder to fix, not easier. A correct recommendation applied at the wrong point in the sequence can do real damage.
Why This Is Worth Knowing
"Nothing's down, but something's wrong" is one of the hardest categories of production investigation, precisely because there's no clear entry point. The temptation is to either dismiss it (nothing's red, so nothing's actually wrong) or to find the first plausible issue and declare victory. Four bugs is not one bug wearing a trenchcoat — but when you're looking for a single explanation, it's tempting to stop at the first one that fits well enough.
Takeaways
- A vague complaint with no red dashboards doesn't mean nothing's wrong — it often means the problem doesn't map cleanly to the alerts you've already built.
- When an investigation turns up multiple issues, resist fixing whichever one is easiest first — sequence fixes by risk and by what's actually causing the reported symptom.
- A "correct" recommendation (like a stricter PDB) can still be the wrong next step if applied before more foundational cleanup is done.