The GitOps Footgun Nobody Warns You About

A colleague manually deleted a Deployment on an app running with ArgoCD's selfHeal: true. On paper that's an unremarkable action — delete a resource, GitOps notices, GitOps fixes it. In practice, that exact combination is more dangerous than it looks, and it cost us a stuck cluster twice before we wrote it down as a rule.

Two Automated Systems, One Object, No Coordination

Here's the race. The moment the Deployment is deleted:

• ArgoCD's reconciliation loop notices the live state no longer matches Git, and — because self-heal is on — immediately tries to recreate the same-named resource via server-side apply.
• Kubernetes' foreground-deletion garbage collector is still working through cascading the original deletion to dependent objects (ReplicaSets, then pods) and hasn't finished yet.

These two processes have no awareness of each other. ArgoCD doesn't know the garbage collector is mid-cascade; the garbage collector doesn't know ArgoCD is about to try recreating the object it's still tearing down. They collide, and the result is a Deployment (and its ReplicaSet, and its pods) stuck mid-termination behind finalizers that never clear on their own — because the object they're attached to is now in an inconsistent state neither system will resolve unilaterally.

The Fix, When It's Already Stuck

If you're already in this state, the fix is mechanical but needs to be done in order. First, confirm what's actually stuck:

kubectl get deployment <name> -n <namespace> \
  -o jsonpath='{.metadata.finalizers}'

If that returns a non-empty finalizer list on an object that's supposedly being deleted (or recreated) and going nowhere, clear it:

kubectl patch deployment <name> -n <namespace> \
  --type=merge -p '{"metadata":{"finalizers":[]}}'

Then force-delete any pods left orphaned in the wreckage:

kubectl delete pod <pod-name> -n <namespace> \
  --grace-period=0 --force

Once the stuck objects are actually gone, let ArgoCD re-sync clean against the Git state. In most cases this fully recovers the app with no further manual intervention.

The Second Time, With a Twist

The same category of problem showed up again later, on a different app, from a different trigger. A git commit removed both app.yaml and values.yaml in the same change. That simultaneous removal meant ArgoCD's own finalizer couldn't render the Helm manifest it needed in order to compute what it should prune — which deadlocked the deletion of the Application object itself, not just a workload underneath it.

Different symptom, same underlying category: GitOps automation and Kubernetes' own object lifecycle stepping on each other in ways neither system is designed to detect or avoid on its own.

Turning It Into a Rule Instead of a Memory

Once this had happened on more than one cluster, treating it as a one-off stopped making sense. The fix that actually sticks isn't "remember not to do that" — under incident pressure, nobody reliably remembers the right sequence of commands for a problem they've hit twice in six months. The fix is a rule:

• Always pause auto-sync before any manual deletion on a self-heal-enabled app:

  argocd app set <app> --sync-policy none

• Prefer kubectl rollout restart over deleting Deployments directly when the goal is just to force a fresh rollout — it doesn't touch the object identity ArgoCD and Kubernetes are both tracking, so there's nothing for the two systems to race over.

Both of those are now written into the team runbook, with the exact commands above, not a paraphrased description of them. When this happens under pressure at 2am, "go read the concept" isn't useful — the actual command needs to be one copy-paste away.

Takeaways

• ArgoCD self-heal and Kubernetes' foreground-deletion garbage collector don't coordinate — manually touching a self-heal-enabled resource is asking two automated systems to fight over the same object.
• Removing multiple files that a Helm chart needs simultaneously (like app.yaml and values.yaml together) can deadlock the deletion of the ArgoCD Application object itself, not just the workload.
• When a production gotcha happens twice, the fix isn't a better memory — it's a runbook with literal, copy-pasteable commands, and ideally a habit (pause sync first) that prevents the race from ever starting.