When you manage a handful of Kubernetes clusters, you can get away with manual processes. When you manage 300+, every inefficiency becomes a crisis. Over the past few years, I worked on a fintech platform that provisioned dedicated Kubernetes clusters for banking clients — each with strict isolation, compliance requirements, and zero-downtime expectations.
Here are the key lessons from that experience.
1. Standardize Everything, Then Automate
The first instinct when managing many clusters is to automate provisioning. But automation without standardization just scales chaos faster. Before writing a single pipeline, we locked down:
- Cluster templates — a base configuration (Terraform modules) that every cluster inherits, covering networking, RBAC, logging, and monitoring
- Kubernetes version policy — a defined upgrade cadence with canary rollouts to a subset of clusters before fleet-wide deployment
- Add-on versions — pinned versions for ingress controllers, cert-manager, external-dns, and monitoring agents across the fleet
Once the baseline was stable, we could automate provisioning with Terraform and Pulumi. A new banking client could go from contract signature to a fully provisioned, production-ready cluster in under a day.
2. GitOps Is Non-Negotiable at Scale
With 300+ clusters, you cannot SSH into boxes or run kubectl apply manually. GitOps was the only way to maintain sanity. We used Flux as the primary reconciler, with ArgoCD providing a UI for visibility into sync states.
The workflow was straightforward:
- All cluster configurations lived in Git repositories, organized by tenant
- Changes went through pull requests with mandatory review
- Flux watched the repos and reconciled cluster state automatically
- Drift detection flagged any manual changes that deviated from the declared state
The critical insight: Git becomes your audit log. Every change to every cluster is traceable to a specific commit, reviewer, and timestamp. For banking clients, this auditability was a compliance requirement, not just a nice-to-have.
3. Fleet-Wide Upgrades Need a Strategy
Upgrading Kubernetes across 300+ clusters is one of the most stressful operations you can perform. We developed a tiered upgrade process:
- Tier 1 (Canary) — internal dev/staging clusters upgraded first. These caught most breaking changes before they hit production
- Tier 2 (Early Adopters) — a small set of lower-risk production clusters, upgraded a week after Tier 1
- Tier 3 (Fleet) — remaining production clusters, upgraded in batches with automated health checks between each batch
Automated health checks were critical. After each batch, we verified: API server responsiveness, pod scheduling, ingress connectivity, and application-level health endpoints. If any check failed, the rollout paused and alerted the on-call engineer.
4. Observability Across the Fleet
Each cluster ran a local Prometheus instance for metrics collection, but aggregation happened centrally. We used Thanos to federate metrics from all clusters into a single queryable data store, with Grafana dashboards that could slice by tenant, cluster, region, or Kubernetes version.
Key metrics we tracked fleet-wide:
- Kubernetes API server latency (p95 and p99)
- Node readiness and pod scheduling latency
- Flux sync status and reconciliation lag
- Certificate expiry (cert-manager)
- Resource utilization vs. requests/limits ratios
Sysdig provided runtime security monitoring, catching container anomalies and policy violations across the fleet.
5. Tenant Isolation Is More Than Namespaces
For banking workloads, namespace-level isolation was not sufficient. Each tenant ran in a dedicated cluster with:
- Dedicated VPCs with private subnets and no cross-tenant network paths
- Separate IAM roles and service accounts per tenant
- Dedicated node groups with instance-level isolation
- Network policies enforcing strict ingress/egress rules
This increased operational complexity but was non-negotiable for the compliance requirements of the banking sector.
Key Takeaways
- Standardize before you automate — scaling bad patterns is worse than manual processes
- GitOps is mandatory for fleet management — it provides auditability, consistency, and drift detection
- Upgrade strategies need tiers, automated health checks, and rollback procedures
- Centralized observability with per-cluster agents gives you both overview and depth
- True tenant isolation requires infrastructure-level separation, not just Kubernetes namespaces
If you are scaling a Kubernetes platform beyond a handful of clusters, investing in these patterns early will save significant operational burden as you grow.