The hidden failure paths that affect Kubernetes control plane availability

Kubernetes control planes are designed to contain individual failures, but their availability still depends on how components in the critical path behave under pressure. At fleet scale, a component does not have to crash to contribute to an outage. A valid mechanism can amplify a local defect through shared metadata, synchronized startup work, or an overly aggressive health check. For example, an informer gives a controller a local view of cluster state, letting its workers make progress without repeatedly querying the API server. This is efficient when controllers start independently. When many controllers restart simultaneously across a fleet, rebuilding those views can place significant pressure on the control plane. The informer mechanism itself becomes the amplifier. “At fleet scale, a component does not have to crash to contribute to an outage. A valid mechanism can amplify a local defect through shared metadata, synchronized startup work, or an overly aggressive health check.” This article examines three cases. In the first, an extension API server published an invalid schema after a dependency upgrade and blocked clients that had never used its API. In the second, DaemonSet agents restarted across a large fleet, turning routine startup work into a request storm the API server couldn’t absorb. In the third, a deep liveness check on a component in the API request path treated a dependency outage as a process failure and drove repeated restarts while the dependency remained unavailable. These cases matter for teams running extension APIs, controllers that watch large object collections, and components in the API request path that rely on deep dependency checks.

1. Extension API Servers: How a Dependency Upgrade Broke Unrelated Clients

In one of the cluster, Helm deployments started failing for charts that contained only standard Kubernetes resources. The charts had no relationship to the extension API that caused the failure. Users who had never heard of the extension saw validation errors before their requests reached the API server. The failure path ran through metadata, through a schema document that combined definitions from APIs registered in the cluster. Understanding why requires knowing how Kubernetes exposes API metadata to clients. Kubernetes can be extended through aggregated API servers, which add new APIs without modifying the core API server. To users, an aggregated API looks native: it has a group, version, kind, and resource paths, and kubectl accesses it through the cluster’s normal API endpoint. An extension API server generates an OpenAPI schema describing its objects and fields. Clients retrieve this schema to understand and validate resources before sending requests.

OpenAPI v2 and v3

Kubernetes exposes this schema contract in two forms, and the difference determines the blast radius of a broken extension. OpenAPI v2 is a single combined document available at /openapi/v2, containing definitions from the core API server and aggregated extension APIs. OpenAPI v3 separates schemas by API group and version. That separation narrows the impact because clients using a core API don’t need to retrieve every extension’s document. “The v2 format creates a shared failure domain: an invalid definition from one extension can prevent clients from parsing the document used to validate unrelated resources.” The v2 format creates a shared failure domain: an invalid definition from one extension can prevent clients from parsing the document used to validate unrelated resources.

Explicit OpenAPI model names

Kubernetes PR #131755 made OpenAPI model names explicit. Before this change, Kubernetes could derive a model name from a Go package path. That made it easy to start with a valid schema, but it also tied published model names to implementation details. Moving code or reorganizing packages could change a name that OpenAPI clients saw. The newer approach lets API authors declare a stable model package, and code generation uses that declared value when building schema definitions and references. Before the change, a schema could expose a definition derived from the raw Go package path:

{
   "com.example.platform.pkg.apis.metrics.v1.MetricReportSpec": {
     "type": "object"
   }
 }
After the change, once the package declares an OpenAPI model namespace, the same definition uses an explicit, stable name:
{
   "com.example.platform.metrics.v1.MetricReportSpec": {
     "type": "object"
   }
 }

A missing declaration

One extension API server upgraded the Kubernetes API machinery dependency without including the +k8s:openapi-model-package annotation required by the updated model name generation. Its generated code continued to supply raw Go package paths when building schema definitions. Earlier API machinery transformed those paths into OpenAPI-safe names; the newer behavior expects the extension to provide a declared model package. Without that declaration, the extension began publishing raw package paths as OpenAPI model identifiers. Because those paths contain /, they made the resulting schema invalid. The extension server started successfully and continued serving its API. Process health checks do not validate the OpenAPI document a process publishes. The problem appeared in clients that read OpenAPI v2. Because v2 combines API schemas into one document, an invalid definition from one extension can cause a client to report an error before it sends the requested API call, even when the requested resource belongs to a different API. Helm’s behavior made this especially visible. Depending on the Helm version, command, and validation settings, Helm retrieves the OpenAPI v2 document before applying a chart. A chart using only ordinary Kubernetes resources can fail because an unrelated aggregated API publishes an invalid schema. Clients using v3 for core resources follow a narrower path: they retrieve the relevant core API document rather than the extension schema that contains the problem.

Test the schema you publish

The most revealing test validates the artifact that clients actually receive. After an upgrade, start the extension and fetch both schema formats:

kubectl get --raw /openapi/v2 > openapi-v2.json
 kubectl get --raw /openapi/v3/apis// > openapi-v3.json

Parse and validate those documents before enabling the extension in production. Also test with clients that validate requests against the OpenAPI v2 schema. These tests follow the user path and reveal problems that endpoint health checks cannot detect.

2. When a rolling restart becomes a request storm

The cost of rebuilding a local view

A DaemonSet runs one copy of a Pod on every node in a cluster. It suits monitoring, network policy, logging, and security agents that need an up-to-date view of cluster Pods. For example, a network agent may need to map an IP address to its owning Pod for policy or telemetry. The standard way to maintain that view is through an informer: a client mechanism that lists the required objects and then watches for changes to keep a local cache current. On a cluster with a few thousand nodes, a cluster-wide Pod informer means a few thousand potential initial list operations. When agents start at different times, the load spreads across the API server’s capacity without incident.

When recovery synchronizes the work

The cost changes when recovery synchronizes the work. In one incident, a DaemonSet ran across thousands of nodes, and its agents repeatedly crashed across much of the fleet. An operator initiated a rolling restart to recover them. The DaemonSet’s maxUnavailable setting seemed protective because it limits how many Pods an update may make unavailable at once. Those Pods were already unavailable. Their replacements did not necessarily face the same pacing constraints. Many agents started nearly simultaneously. Each new agent creates its informer. Each informer begins by listing the same collection of Pods. A recovery operation intended to reduce the failure footprint can therefore create a large, synchronized population of requests against the API server.

The cost of a list request

Informers commonly make their initial request with the resourceVersion parameter set to 0. By using 0, the client allows the API server to answer from its watch cache in memory rather than performing a direct read from etcd, the API server’s backend datastore. This is normally the fast path. It avoids repeated storage reads and uses a view of the state the API server has already assembled for watches. For years, that path had an unintuitive limitation. The watch cache could not provide pagination in the same way that lists served from etcd could. A client could include limit=500 and reasonably believe it had bounded the response to 500 objects. In affected request paths, when the list was served from the watch cache with resourceVersion=0, the API server could ignore the limit and return every matching object. On a large cluster, a response containing all Pods may include tens of thousands of objects and occupy hundreds of megabytes while the API server decodes, filters, serializes, and sends it. When thousands of replacement agents request the same full collection together, the server can exhaust available memory and CPU constructing many copies of the same answer. Ordinary control plane traffic then waits.

Reducing the startup load

The Kubernetes community has improved this part of the stack. API Priority and Fairness list-cost estimation was corrected in Kubernetes 1.31. WatchList can also replace the traditional list-then-watch startup path in clients that support it. These improvements reduce the per-request cost, but they do not change the broader design calculation: a cluster-wide informer in a DaemonSet still multiplies its startup cost by the number of nodes, and a retry policy can still synchronize that cost during a failure. The question for an agent owner is: what is the smallest Kubernetes resource collection this informer can watch and still fulfill its function? An agent that only needs Pods in one namespace should not list all Pods cluster-wide. An agent that needs Pod metadata should consider a metadata-only informer rather than transferring full Pod specifications. An agent interested in service backends may be able to use EndpointSlices instead of listing every Pod because EndpointSlices describe service endpoints directly and are typically much smaller than the full Pod inventory. An agent that truly needs Pod-to-IP ownership, labels, or other Pod metadata may still need to watch Pods, but it should restrict that watch to the relevant namespaces, nodes, or labels whenever its semantics allow. The goal is not to eliminate Pod informers. It is to avoid reconstructing more cluster state than the agent can use, especially when recovery causes many instances to start together.

3. When a liveness check restarts what it cannot fix

A liveness probe that checks downstream dependencies will restart a process whenever those dependencies are unavailable. If the process sits in the API request path, each restart can interrupt API traffic. When the dependency outage persists, the probe drives a cycle of restart, initialize, fail dependency check, and restart again. Each iteration adds latency without repairing the dependency. “The probe may faithfully execute its contract. The contract may still be wrong.” The probe may faithfully execute its contract. The contract may still be wrong.

Static stability depends on cached key material

Kubernetes control planes may depend on an external Key Management Service, or KMS, to encrypt Secrets. The KMS provider uses that service for key operations, while the API server can retain cached encryption material in memory. That cached state is part of the system’s availability design. A transient loss of KMS connectivity may delay operations that require fresh key material without immediately making the A