The Upgrade That Broke Nothing
One app out of twenty went unhealthy. The error named a field nobody had ever set — and by the time we finished, the upgrade that fixed it had lit up two more things that had been quietly broken for years.
The complaint
The Project is a GitOps repository. It drives two Kubernetes clusters — one for development and staging, one for production — and ArgoCD reconciles everything in it.
The monitoring stack stopped reconciling. Everything else was green. The error read:
error calculating structured merge diff: error building typed value from live resource: .status.terminatingReplicas: field not declared in schema
Nobody had written terminatingReplicas anywhere. It isn't a field you set. It's a field Kubernetes reports — the number of pods still shutting down — added to Deployments in 1.33 and switched on by default in 1.34.
So the cluster had grown a field, and the deploy tool refused to read it.
The part that surprises people
The obvious assumption is that ArgoCD asks the cluster what a Deployment looks like. It doesn't.
ArgoCD's own FAQ is blunt about it: it "relies on a static (hard-coded) set of schemas for built-in Kubernetes resource types." That schema is frozen at build time, fixed by whichever Kubernetes libraries that release was compiled against.
cluster (Kubernetes 1.36) ArgoCD (built against 1.32)
───────────────────────── ───────────────────────────
Deployment.status Deployment.status
replicas replicas
readyReplicas readyReplicas
updatedReplicas updatedReplicas
terminatingReplicas ← new · (nothing here)
└─ "field not declared in schema"
Our production ArgoCD was from late 2022. The cluster was current. Nothing was misconfigured — the two had simply drifted four years apart, and the cluster finally said something the tool had no word for.
We spent a little while trying to tell ArgoCD to ignore the field. That doesn't work, and it's worth knowing why: the failure happens while building the object, before there's a diff to exclude anything from. There is no "skip this field" hook early enough. Upgrading is the only real fix.
Why only one app?
This was the detail that made the whole thing make sense.
Twenty-odd applications, same cluster, same new field on every Deployment. One broken.
The broken one was the only app configured with server-side apply. Old ArgoCD quietly picked a different diff strategy for those — one that parses live cluster state into typed objects. Every other app used the default strategy, which never builds a typed object and therefore never trips on an unknown field.
So the blast radius wasn't "apps affected by the Kubernetes upgrade". It was "apps that needed server-side apply" — which, in practice, means apps shipping CRDs too large to apply any other way.
Hold that thought.
The wrong turn
Picking the target version should have been the boring part.
Newer ArgoCD ships a new major version of Helm. Every app in the repo renders a Helm chart. That felt like a real risk, so the plan was to take the previous release train — get the fix, skip the Helm bump, be sensible.
Then someone asked what the clusters were actually running.
They were on Kubernetes 1.36. The "sensible" version is tested up to 1.35. The newer one — the one being avoided — is the only current release tested against 1.36.
The conservative choice was the unsupported one.
That's the beat worth keeping. The hedge wasn't irrational; it was priced against a fact nobody had looked up. A risk you can name and soak-test beats a supported-matrix violation you can't do anything about. Look up the environment's version first, then choose — everything else is preference.
The installer hits its own wall
Installing the new version failed.
Not on our config. On ArgoCD's own manifest, which now contains a custom resource definition too large for a plain kubectl apply — it exceeds the 262144-byte annotation that client-side apply writes to record what it just applied.
The fix is to install with server-side apply.
Which is the same limit, for the same reason, as the one that put server-side apply on the monitoring stack in the first place — the setting that selected the diff strategy that started all of this. The bug and the installation of its fix were the same constraint wearing different hats.
The twist: nothing we "broke" was new
The upgrade landed. The monitoring stack went healthy. And an unrelated app went red.
A CRD belonging to the load balancer controller was reporting a schema violation:
NonStructuralSchema: True spec.preserveUnknownFields: Invalid value: true: must be false
The obvious read is that the upgrade broke it. The obvious read was wrong, and there's a cheap way to prove it: check whether the check existed before. The health check that produced this message is a file in ArgoCD's source tree. Request it at the old version tag and the new one — 404, then 200.
The check was new. The defect was from 2021.
Here's how it got there. That CRD was created from a manifest written against an older API version, where preserveUnknownFields defaults to true. Nobody wrote it. It was implied, and stored.
Every manifest since — the rewritten bootstrap file, the chart's own copy, years of GitOps syncs — simply omits the field. And omitting a field is not the same as removing it. Nothing ever cleared the value, because nothing ever mentioned it.
Worse, the resource read Synced the entire time. A GitOps diff compares what you declared against what's live. A field that exists only in live state isn't in that comparison at all. Sync status is a statement about the fields you wrote — never about whether the object is valid.
Four years green. One explicit patch to set the field to false, and it cleared.
The coda: the jobs that never ran
While checking cluster health, someone noticed a batch of scheduled jobs failing. kubectl get jobs showed failures going back two days.
Two things were false about that.
They hadn't been failing for two days. Kubernetes keeps only the last few failed jobs. When something fails longer than that window, every record you can see is a failure, and the oldest visible one looks like the start. The retained successful runs told the truth: the last success was about ten months earlier. For one job, three years.
They weren't failing — they were never running. Each job died with DeadlineExceeded, which reads like "this took too long". It doesn't mean that. The deadline counts from the moment the job is created, not from when a container starts. The pods couldn't be scheduled, sat pending, and burned the entire budget doing nothing. No container ever ran, which is why there were no logs to read. The timer fired at almost exactly its configured value — punctual in a way real work rarely is.
The cause was one commit, ten months old. A node group had been retired, and the config file was updated to point the workload somewhere else. The file contained eleven scheduling rules: one for the long-running service, ten for the scheduled jobs. The commit changed one of them.
The service moved and stayed healthy. Which is precisely why nobody noticed the other ten were pointing at a node group that no longer existed. The healthy thing was hiding the broken ones.
Hints for the reader
Your deploy tool may not read your cluster's API. Anything with a compiled-in schema is a version dependency you didn't know you had, and it fails on the cluster's new features, not your config.
Look up the version before choosing a version. Hedging against a risk you've named, while the constraint that decides the answer is still unlooked-up, is how "conservative" becomes "unsupported."
Omitting a field never removes it. Especially across an API version migration, where defaults change and manifests get rewritten to stop mentioning things. Set values explicitly when they matter.
Synced is not valid, and green is not healthy. Both are statements about what a tool checks. Neither says anything about what it doesn't.
When an upgrade "breaks" things, date the check before you blame the change. Tools gain checks. New checks grade old work for the first time. A rollback here would have thrown away a working upgrade and re-hidden two genuine defects.
Bounded history lies about duration. Retention windows answer "is it broken now", never "since when". Find the last success, then corroborate against something unbounded — git history usually.
A healthy component can mask a broken migration. When you retire a node group, a queue, or an endpoint, search the whole config for the old name. The thing that still works is not evidence the move finished — it's the reason nobody looks.