Kubernetes disruption budgets during GPU maintenance

Kubernetes can permit an eviction that leaves an inference service short of capacity. A disruption budget counts available pods; a user waits for a request to finish. Between a worker draining and its replacement loading the model, those two measures can tell different stories. The maintenance procedure needs to explain that interval, not just get the drain command to exit successfully.

A useful maintenance rehearsal follows the requests that surviving replicas must carry while a worker is unavailable. Defining the request mix, observation window, stop condition and recovery path lets the team judge that interval against the service it needs to provide. The canary deployment guide applies this test-scope principle to a canary; here the change is temporarily removing a worker. The resulting service behavior is what can justify the next maintenance step.

An eviction allowance is not a reservation for a replacement GPU

A PodDisruptionBudget, PDB, sets availability limits for selected pods during planned disruption. The Eviction API checks it before permitting removal. It does not reserve hardware for a replacement and cannot prevent involuntary failures, although those failures count against the budget. Application rolling updates are governed by the workload controller rather than limited by the PDB. The Kubernetes disruption model explains these boundaries.

That leaves a coordination problem if host maintenance and an application rollout overlap. Each controller can follow its own policy while reducing the same service’s usable capacity. Pause unrelated rollouts during the pilot, identify other maintenance automation and record the exclusion window. The result then describes the change being tested rather than several simultaneous reductions.

Consider a hypothetical service with three Ready replicas, each able to sustain 20 requests per second within the agreed latency limit, facing demand of 50. A two-pod availability requirement can allow one replica to leave, yet the remaining assumed capacity is only 40 requests per second. The extra 10 must wait, be rejected or go elsewhere until capacity returns. These numbers are illustrative, not measured inference performance; prompt length, output length and batching change the real capacity. They show why the replacement interval needs a request-level test.

The workload and spare GPU capacity in staging

Use staging with production-like model size, startup path and scheduling constraints. Inspection of the workload, PDB and placement needs appropriate access, and the drain needs separate authorization. Agree on traffic mix, latency objective, rejection ceiling, maximum observation window and recovery owner first. Synthetic prompts should represent permitted request sizes without copying customer content.

Start with an endpoint whose replicas serve independently. A distributed model whose cooperating ranks answer one request is a different unit, as is a training job with valuable uncheckpointed work. If losing one process is intolerable, use the workload’s checkpoint or coordinated shutdown mechanism. A count-based replica budget cannot invent that recovery capability.

Include the experiment’s resource cost in the plan. Spare GPUs, model downloads, warmup requests and retained logs may incur charges; bound traffic and temporary capacity. If the procedure relies on a spare GPU, identify its funding and actual acquisition time. Capacity that exists only for an unusually well-provisioned test may not support a production maintenance window.

Match the budget to the pods you mean to remove

This example assumes an existing three-replica Deployment labeled app: inference-api in namespace inference. It permits at most one unavailable replica in that selected population; it does not create or scale the application. Inspect live template labels before treating the manifest as the intended protection:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: inference-api-maintenance
  namespace: inference
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: inference-api

Use either minAvailable or maxUnavailable. Integers can make a small population’s allowance easier to inspect, while percentages round up and may allow more disruption than expected. In policy/v1, an empty selector matches all namespace pods. The configuring a PDB documentation explains these choices and the resulting status fields.

kubectl -n inference get deployment inference-api -o yaml
kubectl -n inference get pods -l app=inference-api -o wide

Save the reviewed configuration as inference-pdb.yaml. In the approved staging context, examine kubectl diff -f inference-pdb.yaml and use kubectl apply -f inference-pdb.yaml only when the target and allowance match the plan. Then inspect the applied budget before the drain, so the maintenance decision uses current observations rather than merely the file just applied:

kubectl -n inference get pdb inference-api-maintenance -o yaml

Read currentHealthy, desiredHealthy and disruptionsAllowed together and ensure status reflects the current specification. A zero allowance is a condition to investigate: an unhealthy replica, incorrect selection or a replacement unable to become Ready may explain it. Removing the PDB would remove the objection without providing the missing healthy capacity.

Placement and model loading delay the replacement

A replacement needs an eligible node, not just aggregate free cluster resources. GPU type, resource requests, taints, affinity, storage attachment and topology can each narrow its destinations. Inspect that path before removing capacity. The node-assignment documentation describes the constraints the scheduler must satisfy, helping establish whether the replacement assumed in the maintenance plan can actually be placed.

Current placement matters as well. Two replicas sharing the drained worker do not independently survive its removal. A one-pod disruption allowance might permit the first eviction and block the next until a replacement is healthy. That can be the protection working correctly, while still making the maintenance interval longer than planned.

Measure scheduling, image retrieval, model loading, warmup and the first useful representative request separately. A Running process is not necessarily ready to absorb inference work. Readiness should reflect what can be served without creating uncontrolled expensive inference probes. The probe documentation distinguishes startup, liveness and readiness responsibilities.

Requests, waiting and rejection during the drain

Establish a baseline with the agreed request mix and offered load, including requests the service may reject. Record accepted completions, latency, rejection, queue age and Ready replicas. Keep request identity across retries so multiple attempts do not become fictitious additional completed work. Record load-generator limits too; a bottleneck in the test can hide the shortage being measured.

Once the baseline and target are checked, drain the authorized staging worker using its verified name in place of the placeholder:

kubectl drain <approved-staging-node> --ignore-daemonsets --timeout=10m

The timeout limits command waiting, not the effects already performed. An error exit can follow successful evictions. The documented drain procedure honors the PDB and uses graceful eviction, while DaemonSet pods remain. A successful drain therefore does not mean an empty node, and a failed command does not mean unchanged placement.

Adding --disable-eviction would switch to deletion and bypass the PDB, changing the test rather than diagnosing a stall. --delete-emptydir-data allows loss of local emptyDir data, while --force allows removal of pods without a controller. Read the command reference before considering these flags; each changes the consequences that require authorization.

At the same offered load, compare completions, waiting and rejected work before allowing another node to leave. If the agreed stop condition triggers, halt further maintenance and preserve events, scheduling reasons and service measurements. Determine whether capacity was lost through eviction, slow startup, placement failure or insufficient surviving replicas. A lower served request rate caused by rejection must remain visible rather than being counted as recovery.

Returning a node to scheduling does not undo the drain

If the original worker is healthy and the maintenance owner permits its return, kubectl uncordon <approved-staging-node> allows scheduling again. It does not put evicted pods back, reverse a host update or move existing pods. Recheck placement and request behavior; if the host is suspect, restore capacity through the approved replacement path instead.

An unhealthy pod blocking drain may require an explicit unhealthy-pod eviction policy. Kubernetes recommends considering AlwaysAllow for misbehaving applications. Review that choice with the application owner and cluster version: it trades waiting for recovery against permitting eviction of an unhealthy running pod. Treat it as a decision about this workload rather than a flag to paste into every PDB.

The maintenance worksheet keeps selected pods, baseline, allowed interruption, recovery and next decision together. Promote only the conditions actually exercised. If success depended on spare GPUs unavailable in production, solve the capacity or placement constraint before widening the drain procedure.

Reconstruct the interval from eviction to the replacement’s first useful request, with completions, waiting and rejections beside it. That timeline explains whether two surviving Ready pods were enough while the third returned. If they were not, the pilot has still delivered a useful result: a specific capacity or placement problem to fix before another production worker leaves service.