Kubernetes 1.37: What Breaks on Upgrade Day (and Why It's Your Control Plane)

Kubernetes v1.37 “Garhwal” shipped on 26 August 2026, and the headline breaking change is being reported wrong almost everywhere. v1.37 does not introduce the rule that static Pods cannot reference API objects — that landed in v1.34. What v1.37 removes is the escape hatch. The PreventStaticPodAPIReferences feature gate is gone, and per the upstream release note it “cannot be disabled anymore.”

That inverts who is at risk. If you never hit this, you are fine. The clusters that break on upgrade day are precisely the ones that already hit it in 1.34, 1.35 or 1.36, turned the gate off to get moving, and filed the real fix under later. Later is now, and the thing that fails to start is the control plane.

The three blockers, ranked

Everything else in this release — kube-dns deprecated in favour of CoreDNS, kubectl run --filename/-f deprecated, sixteen features graduating to stable — is housekeeping. The two that stop a node from coming back are the first two.

1. Static Pods can no longer reference API objects

What the rule actually is

The clean formulation is not “no Secrets or ConfigMaps.” It is the one from the implementing pull request: static Pods may only use hostPath and emptyDir volumes, and may not reference API objects at all.

The upstream release note for PR #131837, which carries an explicit ACTION REQUIRED, names the full surface:

Prior to upgrade, ensure static pods are not referencing API objects such as ServiceAccounts, ConfigMaps, Secrets, ResourceClaims, CSIDrivers, PersistentVolumeClaims, or ClusterTrustBundles.

Note ServiceAccounts in that list. Most of the secondary coverage of this release only mentions configMapRef and secretRef, which understates it: a static Pod with a serviceAccountName, an imagePullSecrets entry, or a PVC-backed volume is equally denied. In practice the fields to hunt for are:

• env.valueFrom.configMapKeyRef and env.valueFrom.secretKeyRef
• envFrom.configMapRef and envFrom.secretRef
• volumes[].configMap, volumes[].secret, volumes[].persistentVolumeClaim, volumes[].projected
• serviceAccountName, imagePullSecrets, resourceClaims

Why this is a control-plane problem, not a workload problem

Static Pods are the ones the kubelet runs straight off disk from /etc/kubernetes/manifests/, with no API server involved. On a kubeadm cluster that directory holds etcd, kube-apiserver, kube-controller-manager and kube-scheduler.

So the failure mode is not “one of my Pods didn’t schedule.” It is: you upgrade a control-plane node, the kubelet refuses admission to the etcd manifest, and the API server never comes up to tell you why. You are debugging from journalctl -u kubelet on the node, not from kubectl.

The old behaviour was silent, which is why this exists

Before v1.34 this did not error. The static Pod ran, and only the mirror Pod — the read-only API representation the kubelet creates so the Pod shows up in kubectl get pods — failed to reconcile. The container was live on the node and effectively invisible to the API. PR #131837 closed that by denying admission outright, so the container is never created rather than running unobserved.

That history is the reason the gate existed at all: reviewers asked for the validation to ship on by default but disableable “for a few releases.” Three release cycles later, PR #140226 removed it, with milestone v1.37.

The audit, before you touch a single node

Run this on every control-plane and worker node, not just one:

grep -rlE 'configMapRef|secretRef|configMapKeyRef|secretKeyRef|serviceAccountName|imagePullSecrets|persistentVolumeClaim' \
  /etc/kubernetes/manifests/

Any file it prints is a manifest you have to rework before upgrading that node. The fix is always the same shape: stop pulling the value from the API and put it on disk, as a hostPath mount or as a literal in the manifest, since hostPath and emptyDir are the only volume types still permitted.

If your manifests are generated — kubeadm, Cluster API, a Helm chart that templates node config, an Ansible role — audit the template, not just the rendered output, or the next node you provision reintroduces the problem.

Before you commit a reworked manifest, sanity check that it is still valid YAML and that you did not break indentation while ripping out a volume block:

2. cgroup v1: the kubelet just exits

This one is short and absolute. The kubelet refuses to run on a host using cgroup v1. It does not warn and degrade — it exits on start with:

kubelet is configured to not run on a host using cgroup v1

The default flipped to failing in v1.35, and v1.37 keeps it enforced. The only override is in KubeletConfiguration:

failCgroupV1: false

Where you put it depends on your distribution, and the timing matters:

• kubeadm: edit the kube-system/kubelet-config ConfigMap and add the field before you upgrade. Adding it after the kubelet has already refused to start means editing config on a node whose control plane may be down.
• RKE2 / K3s: pass the kubelet argument fail-cgroupv1=false in /etc/rancher/rke2/config.yaml or /etc/rancher/k3s/config.yaml.

Treat that as a bridge, not a fix. It is the same category of decision as the static-Pod gate you are reading this post because of — an off switch that upstream is winding down.

Check which cgroup version a node is on before you upgrade it:

stat -fc %T /sys/fs/cgroup
# cgroup2fs -> v2, you are fine
# tmpfs     -> v1, you are not

ls /sys/fs/cgroup/cgroup.controllers   # exists only on v2

The nodes that trip this are the old ones: long-lived CentOS 7 and Ubuntu 18.04 hosts, and anything provisioned from an AMI or image built years ago and never rebuilt. If you run a managed service (EKS, GKE, AKS) on current node images you are almost certainly on v2 already — but “almost certainly” is a stat away from certain.

3. kube-proxy IPVS: deprecated, not removed

mode: ipvs in KubeProxyConfiguration is deprecated as of v1.37 (KEP-5495). Nothing breaks today; you get warnings in the kube-proxy startup logs. The timeline:

The rationale is worth knowing because it kills the usual objection: IPVS mode was never a full replacement for iptables — it still requires iptables underneath and cannot fully implement Kubernetes Services on its own. The successor is nftables, not a return to iptables.

Check what you are running:

kubectl -n kube-system get cm kube-proxy \
  -o jsonpath='{.data.config\.conf}' | grep mode:

If it says ipvs, you have from now until v1.40 to move to mode: nftables. That is roughly a year of releases — enough to do it on your own schedule, which is exactly why you should put it on the schedule now instead of meeting it the way people are meeting the static-Pod gate this month.

The pre-upgrade checklist

Run all of this before the first node, on every node:

# 1. Static Pod API references — the upgrade blocker
grep -rlE 'configMapRef|secretRef|configMapKeyRef|secretKeyRef|serviceAccountName|imagePullSecrets|persistentVolumeClaim' \
  /etc/kubernetes/manifests/

# 2. cgroup version — the other upgrade blocker
stat -fc %T /sys/fs/cgroup

# 3. kube-proxy mode — plan, not block
kubectl -n kube-system get cm kube-proxy \
  -o jsonpath='{.data.config\.conf}' | grep mode:

# 4. Are you still on kube-dns instead of CoreDNS?
kubectl -n kube-system get deploy | grep -E 'kube-dns|coredns'

Empty output from 1, cgroup2fs from 2, and nftables or iptables from 3 means the upgrade is boring. That is the goal.

Who actually needs to act

You disabled PreventStaticPodAPIReferences at any point since v1.34: you are the target audience for this entire post. That gate no longer exists, so whatever made you disable it now blocks the upgrade. Fix the manifests first, upgrade second.

You run kubeadm control planes you have hand-edited: run the grep on the control-plane nodes specifically. Hand-edited /etc/kubernetes/manifests/ files are where secretRef and serviceAccountName get added by someone solving a problem at 2am.

You run nodes older than about three years: check cgroups before anything else. A kubelet that exits on start looks identical to a broken upgrade, and you will lose an hour finding a one-line answer.

You are on managed Kubernetes with current node images: you are probably clear on 1 and 2. Check 3 and put the IPVS migration in a quarter, not a sprint.

You run mode: ipvs: nothing breaks now. Schedule the move to nftables before v1.40.

Further reading:

• What breaks upgrading to Node 26 — same format for the runtime, and the same pattern of failures that produce no error.
• What breaks upgrading to Vite 8 — the version you inherited without choosing it, which is the other way these upgrades bite.
• GPT-6 Astra for developers — if you are running agents with shell access against a cluster, the safety and approval-policy section applies directly.

Frequently asked questions

What breaks when upgrading to Kubernetes 1.37?

Two changes can block the upgrade outright. First, the PreventStaticPodAPIReferences feature gate was removed, so static Pods that reference API objects are denied admission by the kubelet with no way to opt out — and on kubeadm clusters those static Pods include etcd and kube-apiserver. Second, the kubelet refuses to start on hosts using cgroup v1 unless failCgroupV1 is explicitly set to false. A third change, the deprecation of kube-proxy IPVS mode, only produces warnings in v1.37 and does not break anything yet.

Why do static Pods fail after upgrading to Kubernetes 1.37?

Because the escape hatch was removed, not because the rule is new. The restriction on static Pods referencing API objects shipped in v1.34 behind the PreventStaticPodAPIReferences feature gate, which was enabled by default but could be disabled. Kubernetes v1.37 removed that gate entirely, and the upstream release note states it cannot be disabled anymore. Clusters that turned the gate off to defer the fix are exactly the ones that break on upgrade.

Which API objects can a static Pod not reference in Kubernetes 1.37?

The upstream release note lists ServiceAccounts, ConfigMaps, Secrets, ResourceClaims, CSIDrivers, PersistentVolumeClaims and ClusterTrustBundles. The simplest way to hold the rule is from the implementing pull request: static Pods may only use hostPath and emptyDir volumes. That means serviceAccountName, imagePullSecrets, envFrom.configMapRef, envFrom.secretRef, env.valueFrom.secretKeyRef, projected volumes and PVC-backed volumes are all rejected, not just the ConfigMap and Secret references most summaries mention.

How do I check whether my cluster is affected before upgrading?

On every control-plane and worker node, run: grep -rlE 'configMapRef|secretRef|configMapKeyRef|secretKeyRef|serviceAccountName|imagePullSecrets|persistentVolumeClaim' /etc/kubernetes/manifests/ — any file it prints must be reworked before that node is upgraded. Separately run stat -fc %T /sys/fs/cgroup on each node; cgroup2fs is safe and tmpfs means cgroup v1, which stops the kubelet from starting. If your manifests are generated by kubeadm, Cluster API, Ansible or Helm, audit the template as well or newly provisioned nodes will reintroduce the problem.

Is kube-proxy IPVS mode removed in Kubernetes 1.37?

No. It is deprecated in v1.37 under KEP-5495 and only logs warnings at startup. It is expected to be disabled by default in v1.40, where it can still be re-enabled through a feature gate, and removed entirely in v1.43. The replacement is nftables mode, not iptables. The reasoning is that IPVS mode always required iptables underneath and could never fully implement Kubernetes Services on its own.

What happened to static Pods referencing Secrets before Kubernetes 1.34?

It failed silently, which is why the restriction exists. The static Pod would run on the node, but the mirror Pod — the read-only API representation that makes the Pod visible to kubectl — failed to reconcile. The container was live and effectively invisible to the API server. PR #131837 changed this to deny admission outright so the container is never created, rather than running unobserved.

When was Kubernetes 1.37 released and what is it called?

Kubernetes v1.37, codenamed "Garhwal", reached general availability on 26 August 2026. Alongside the deprecations it graduated sixteen features to stable, including ResourceClaim device status (KEP-4817), device taints and tolerations (KEP-5055), resource health status for Pods (KEP-4680) and SELinuxMount with SELinuxChangePolicy (KEP-1710). It also deprecated the kube-dns subproject in favour of CoreDNS and the --filename flag on kubectl run.