A GitOps-managed application was stuck running an old container image, despite the correct new version having been committed to the source repository well over half a day earlier. In a properly working GitOps setup, that delay shouldn't exist — reconciliation should have picked up the change within minutes. Something was actively preventing it.
The Theories That Looked Right and Weren't
The first suspect was the values file — and it wasn't a dead end exactly, I found a real structural YAML issue in it and fixed it. But fixing it didn't change the symptom at all. The application stayed on the old image. Real bug, wrong bug.
The second suspect was the automation controller's reconciliation state — maybe it had a stale timestamp and thought it was already in sync. Also real: the timestamp was in fact stale. Also not causal — forcing a fresh reconciliation didn't pull the new image either.
Two real, confirmed issues, and neither one explained the actual symptom. That's the point in an investigation where it's worth stepping back rather than continuing to poke at the current configuration for a third theory.
The Answer Wasn't in the Config — It Was in the History
Looking at commit history instead of current state turned up the actual cause: a commit from the previous week had deliberately disabled the automatic image-update mechanism for this specific application. Someone had intentionally paused automatic deployments to that environment — a completely reasonable thing to do at the time, for reasons that made sense a week earlier and had simply been forgotten by the time this investigation started.
With that auto-update flag off, a separate system — one that writes resolved version numbers as parameter overrides directly onto the deployment object — had quietly stopped updating that parameter. And those parameter overrides take priority over whatever the values file says. Permanently. Until someone either flips the flag back on or manually overrides the parameter by hand. The values file could say anything; it was never going to matter while that override was sitting there frozen from a week ago.
Why This Was Genuinely Hard to Find
Every tool for inspecting "what's configured right now" — the values file, the live deployment object, the controller's reconciliation status — showed a perfectly self-consistent, if stale, picture. Nothing in any snapshot of current state was contradictory or obviously broken. The actual cause was a decision made a week earlier, not a state visible in the present. Snapshots don't show decisions; history does.
Takeaways
- When current-state debugging keeps producing real-but-non-causal findings, check what changed recently — not just what's configured now.
- Parameter overrides that take priority over a values file are a common GitOps pattern, and also a common place for "the file says X but nothing changes" confusion — know which mechanism actually wins for your setup.
- A deliberate, reasonable decision from last week (pause auto-updates) can look identical to an unexplained bug this week, if nothing records why alongside what — a commit message carrying that context is worth the extra sentence.