Kubernetes disruption budgets: test the capacity left behind

A node drain can obey its disruption budget and still leave an inference service short of capacity. Kubernetes has permitted the eviction. The two remaining replicas now have to absorb the requests that previously reached three, while a replacement finds a GPU, loads its model and becomes useful.

A maintenance pilot should exercise the traffic that will remain after an eviction, with an explicit stop condition and a usable recovery path. This applies the test-scope principle in our canary deployment guide to a different change: removing a worker node. The useful result is a maintenance decision backed by service behavior, not simply a completed command.

What a disruption budget controls

A PodDisruptionBudget, or PDB, describes how many selected application pods must remain available during voluntary disruptions that use the Eviction API. It constrains eviction admission; it does not reserve replacement hardware. Kubernetes documents that involuntary failures cannot be prevented by a PDB, although they count against the budget. Application rolling updates are governed by the workload controller rather than limited by the PDB. See the Kubernetes disruption model.

That distinction matters during overlapping work. A host-maintenance controller and a deployment controller can each follow their own policy while reducing the same service's usable capacity. In the pilot, pause unrelated application rollouts and identify other node-maintenance automation. Record the exclusion window so the result describes one change.

In a hypothetical inference service, three Ready replicas each sustain 20 requests per second within the agreed latency limit. Demand is 50 requests per second. Allowing one replica to leave preserves two Ready pods but only 40 requests per second of the assumed service capacity. The remaining 10 requests per second must queue, be rejected or go elsewhere. These are illustrative numbers, not measured model performance, and real inference capacity depends on prompt length, output length and batching.

Prepare a service-sized pilot

Use a staging service with the same scheduling constraints, model size and startup path as production. You need permission to inspect the workload, its PDB and node placement, plus separately authorized node maintenance. Agree on the traffic mix, latency objective, rejection ceiling, maximum observation window and recovery owner before executing a drain. Synthetic prompts should resemble permitted request sizes without copying private customer content.

Start with one independently serving, replicated endpoint. A distributed model whose ranks cooperate to answer one request is not equivalent to several independent replicas. Nor is a long training job with valuable uncheckpointed progress. If a workload cannot tolerate losing one process, use its own checkpoint or coordinated shutdown procedure; a replica budget cannot supply that capability.

Check the cost of the experiment as well as its duration. Spare GPUs, model downloads, warmup requests and retained diagnostic logs may all incur charges. Put a cap on test traffic and temporary capacity. A maintenance plan that depends on spare capacity should name who pays for keeping it available and how long acquiring it actually takes.

Make the selected population explicit

The following example is for an existing three-replica Deployment whose pod template carries app: inference-api in the inference namespace. It permits at most one unavailable replica for that selected population. Confirm the labels against the live Deployment before applying anything; this file does not create or scale the application.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: inference-api-maintenance
  namespace: inference
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: inference-api

Choose either minAvailable or maxUnavailable, not both. For small populations, integer values make the intended allowance easier to inspect. Percentage values round up, which can permit more disruption than a casual reading suggests. An empty selector in policy/v1 matches all pods in the namespace. These details and the status fields are documented in configuring a PDB.

kubectl -n inference get deployment inference-api -o yaml
kubectl -n inference get pods -l app=inference-api -o wide
kubectl -n inference get pdb inference-api-maintenance -o yaml

Read currentHealthy, desiredHealthy and disruptionsAllowed together. Check that the status has observed the current specification. A zero allowance is evidence to investigate, not a reason to delete the budget. There may already be an unhealthy replica, a mismatched selector or a replacement that cannot become ready.

Save the reviewed manifest as inference-pdb.yaml. In the approved staging context, inspect kubectl diff -f inference-pdb.yaml, then apply it with kubectl apply -f inference-pdb.yaml only if the target and allowance match the agreed plan. Re-read the PDB status before the drain.

Inspect the replacement path before removing capacity

Find the nodes eligible for a replacement and explain why each qualifies. GPU type, requested resources, taints, node affinity, volume attachment and topology constraints can leave an apparently roomy cluster with nowhere to place this particular pod. Kubernetes' node-assignment documentation describes the constraints the scheduler must satisfy.

Inspect actual placement too. Two replicas on the same worker do not provide two independent answers to that worker's maintenance. A one-pod allowance may let the first eviction proceed and then block the next until a replacement becomes healthy. That can be protective, but the maintenance window must allow for it.

Model startup makes the timeline important. Record scheduling delay, image retrieval, model loading, warmup and first successful representative request separately. A replacement that is Running but not useful cannot absorb the missing work. Design readiness checks around what the process can actually serve; do not send expensive full inference requests at an uncontrolled probe frequency. Kubernetes distinguishes startup, liveness and readiness in its probe documentation.

Run one drain and retain the evidence

First collect a baseline at the agreed offered load. Include accepted completions, latency, rejected requests, queue age and currently Ready replicas. Keep requests identifiable across retries so a client retry storm is not counted as increased useful throughput. Record the load generator's own limits.

Then drain one approved staging worker, substituting its verified name for the placeholder:

kubectl drain <approved-staging-node> --ignore-daemonsets --timeout=10m

The example timeout bounds how long the command waits; it is not a rollback. Some evictions may already have happened when the command exits with an error. The drain procedure uses graceful eviction and honors the PDB. DaemonSet pods remain, so successful completion does not mean the node contains no pods.

Do not add --disable-eviction to make a stalled pilot finish: that selects deletion and bypasses the PDB. Likewise, --delete-emptydir-data explicitly permits loss of local emptyDir data, and --force allows removal of pods lacking a controller. Their meanings are documented in the command reference. Each changes what the experiment is authorized to do.

Compare completed requests, latency and rejections at the same offered load before authorizing the next node. If the stop condition fires, halt further maintenance and preserve events, scheduling reasons and the service measurements. Investigate whether the shortage came from eviction, slow startup, placement or the surviving replicas' capacity. A lower request rate caused by rejected traffic must remain visible in the report.

Recovery must restore useful service

If the original node remains healthy and the maintenance owner approves returning it, kubectl uncordon <approved-staging-node> permits scheduling again. It does not recreate an evicted pod on that node, undo a host update or move existing pods back. Recheck placement and service behavior; if the node is suspect, restore capacity through the approved replacement path instead.

A blocked drain involving unhealthy pods may require a deliberate unhealthy-pod eviction policy. Kubernetes recommends considering AlwaysAllow for misbehaving applications during drain. Review that choice with the application owner and your cluster version, rather than adding it as an unexplained universal default. The decision trades waiting for recovery against allowing an unhealthy running pod to be evicted.

Use the maintenance worksheet to preserve the selected population, baseline, permitted interruption, observed recovery and next decision. Promote the procedure only for the tested workload and capacity conditions. If the only successful run required spare GPUs that production cannot obtain, the next action is a capacity or scheduling change, not a broader drain.