How Kubernetes Memory QoS affects inference under pressure

The inference process stops restarting, yet people wait longer for answers. Keeping it alive helped one symptom without necessarily helping the service. Kubernetes Memory QoS offers additional ways to manage memory pressure, so the comparison needs to follow what requests experience while the process survives.

The September 14 announcement promotes Memory QoS to beta in Kubernetes v1.37 and enables its feature gate by default on the relevant Linux cgroup v2 path. Default kubelet configuration still enables neither throttling nor reservation; those require separate opt-in. The feature gate alone does not describe the node's behavior.

Separate host RAM from GPU device memory first. Memory QoS uses Linux's host-memory controller. It cannot fit an oversized model into GPU memory, but competition for host RAM around the process can still lengthen a request.

Linux groups processes into control groups, or cgroups, to account for and control resources. Their memory settings influence what happens as a container uses more RAM. Memory QoS changes pressure and protection for those groups; it does not add physical memory. Neighboring work and the actual settings therefore matter together.

Why similar node templates can behave differently

Give the waiting process a concrete memory budget: a 256 MiB request and a 1 GiB limit. The request helps Kubernetes account for the resources it needs; the limit is a separate upper bound. This example uses only container-level resource settings, with no Pod-level resource settings configured. Its memory request below its limit puts the Pod in the Burstable class. Under this container-level route, Guaranteed requires matching positive CPU requests and limits and matching positive memory requests and limits for every container. With throttling enabled at factor 0.9, the threshold is request + factor × (limit − request), roughly 947 MiB here. An allocation above that threshold can leave the request waiting for memory reclaim even before the hard limit is reached.

The new default memoryThrottlingFactor: null means kubelet does not set memory.high. So two upgraded nodes that once inherited the same default can diverge if one now explicitly retains 0.9 and the other leaves the value unset. The first still enables throttling; inheriting the old default is not the same as explicitly preserving it.

The other resource classes change the calculation. BestEffort Pods have no CPU or memory requests or limits, so their containers use a zero memory request for this threshold. Without a container memory limit, the calculation uses node allocatable memory, the node’s budget available for Pods rather than all physical RAM. Guaranteed containers do not get this threshold. These distinctions tell us which policy could be slowing the allocation, not whether the user’s wait is acceptable.

To explain the delayed allocation, look at what the node actually received. Management tooling may have written a value absent from the repository template, or changed an explicit one. The effective values before and after the rollout tell us whether this comparison really changed throttling, rather than merely changing a file that never reached the kubelet.

The request-level cost of keeping a process alive

The QoS documentation distinguishes memory.high throttling from memory.min and memory.low protection. As the Linux documentation explains, exceeding memory.high can slow allocations while reclaim runs. That can delay a request without terminating the process. The hard limit remains separate, so throttling is no promise against exhaustion.

Picture a request ready to continue except for one more allocation. Slowing it gives Linux time to reclaim space, but the user still waits. That delay may fit a background job and violate an interactive deadline. Test both kinds of work against their own commitments and retain the pressure conditions beside the result.

In a hypothetical trial, restarts fall while requests wait longer. The platform change was applied, but its service benefit remains unsettled. Count successful responses within the latency objective to see whether useful work improved.

Tiered reservation also operates across the node's pods rather than as individual pod opt-in. Protected memory can include page cache, where Linux retains file contents. Include neighboring workloads in the comparison; the same serving pod can face different competition beside them.

Survival and responsiveness are different results

Read diagram description

A conceptual comparison under memory pressure. Count restarts and request behavior together; fewer terminations do not by themselves establish better inference service. Diagram labels: Memory pressure: Same representative workload and host constraints; Container behavior: Throttling and restarts; Request behavior: Latency, errors and completions; Compare with baseline: Keep the policy only if the service tradeoff is acceptable.

Comparing serving behavior under memory pressure

Follow the waiting request through a controlled comparison. Before changing policy, decide what counts as useful completion, such as a successful response within the service’s latency objective. Keep the model, request mix, concurrency and neighboring work steady, with the node settings recorded, then change one policy. Otherwise a quieter neighbor or shorter answer could receive credit for an apparent improvement.

Suppose the process survives the pressure episode and begins streaming sooner than in the baseline trial, but finishes after the user’s deadline. The first-token wait improved while the full response still arrived too late. Compare both waits alongside memory pressure and terminations, keeping output length visible so a longer answer is not mistaken for slower resource handling. For retrieval or preprocessing, follow the completed job and any retry effort instead. These are different ways of asking what the extra waiting cost.

A short synthetic pressure episode can show a working configuration and its immediate tradeoff, not what happens after a day of cache growth. Larger reads or changed dependencies may give the same process a different memory burden. If the first comparison succeeds, a bounded rollout needs to examine those conditions before we extend its conclusion.

Use the SLO guide to connect observations to the user's promise and the Linux tuning guide for the surrounding resource investigation. Reducing terminations, reducing allocation wait and meeting response deadlines can suggest different choices.

Rollback includes kubelet configuration

The announcement also warns that disabling the feature needs compatible kubelet settings. Reversing the gate alone may leave configuration to repair. Rehearse the release-specific reversal outside production, then compare effective state with the baseline.

If the late streaming response triggers reversal, the next operator needs more than “turn Memory QoS off.” The node version and cgroup v2 status identify the path under test; effective throttling and reservation values show the state to compare with the baseline. Keep those with the workload mix, service objective and observed result, and identify the reversal trigger and authorized owner. That lets the operator restore the intended configuration and check the request again, rather than assume a reversed flag restored the earlier behavior.

At the end of the pressure episode, the process is still alive and the user has finally received an answer. Was it within the promised time? If fewer restarts came with more on-time work, that supports keeping the tested policy. If fewer useful requests finished, the surviving process did not make the service better. The response’s completion time and the effective kubelet settings belong in the same record, so the next decision can be repeated or reversed.