Find the wait before changing a Linux setting

More reports finish each minute, yet people running small queries wait longer. In a hypothetical shared connection pool, adding workers lets long reports hold more connections. The benchmark improves while one group’s experience gets worse.

Linux tuning starts with the wait you want to reduce: CPU time, storage, a connection or another resource. A setting is useful when it tests an explanation for that wait. Making a number larger because a checklist suggests it gives you no such explanation.

Keep the workload with the baseline

Start with the people waiting for the small queries. Compare their latency distribution in a healthy interval with the interval after more report workers were added. Keep throughput and errors beside that view, and record the mix of long reports and small queries. A busier afternoon is not the same experiment as the same afternoon with one setting changed.

The machine matters too. Keep the kernel, machine type, container limits and recent changes with those observations. The next engineer needs to know whether the concurrency value helped inside this quota or on a different machine. If the quota or workload later changes, the old result may no longer describe the service.

In the same interval, these read-only tools can help narrow where the small query waits. They require the relevant local packages; fields and availability vary:

uptime
vmstat 1 5
iostat -xz 1 5
free -m
ss -s

Read the output against its clock and meaning, using the local manuals. The first vmstat or iostat report may cover time since boot, not the short interval in which the query waited. Load average is not a CPU percentage, and low free memory can include reclaimable cache, so available memory is the more useful next figure to inspect. None of those first glances, by itself, identifies the connection the query needs.

Brendan Gregg’s USE-method checklist groups resource evidence into utilization, saturation and errors. Use it to choose the next question about this wait, not to give every counter a turn on the screen.

Let one observation lead to the next

Suppose the host still has idle CPU while a small query is delayed. In our shared-pool example, that is quite compatible with waiting for a connection held by a long report: a waiting request need not keep a processor busy. It is also compatible with a container quota or another wait, so the idle reading is a clue, not the answer.

The vmstat manual defines the fields that help separate these possibilities. r counts runnable processes, including those waiting for run time; b counts processes waiting for I/O; id is idle CPU time. Persistently high runnable work with little idle time would point toward demand and scheduling instead. Even then, it would not name the function to optimize.

Follow the small query beyond the host chart. Where does its latency accumulate, and what does its trace show before it gets a connection? If blocked work points toward I/O, follow device latency and limits rather than declaring storage the cause. Another command earns its place when it distinguishes the remaining explanations for this request.

Inspect the boundary that limits this process

The host's spare CPU belongs to the whole machine, not necessarily to the process handling our query. A cgroup can restrict that workload to part of the host's capacity. Before concluding that CPU is irrelevant, check the quota the waiting process actually receives. Raising a host-wide setting would not remove a narrower limit that still applies to it.

The same distinction matters for open files. A process can exhaust its own allowance before reaching the host's limit. Inspect that process rather than treating the machine's larger number as room it can use. The cgroup v2 guide describes its interfaces; v1 paths should not be copied blindly into v2. On a managed platform, use the supported workload configuration instead of editing controls behind the orchestrator.

If the query's wait leads to storage, ask what is keeping that operation there: queueing, a volume limit or the access pattern. A scheduler that suited an older kernel may be unavailable or wrong for the present device. In the shared-pool case, changing that familiar setting would be a detour unless the observations connect it to the long reports holding the connections.

Test the effect on both kinds of request

Write the expected change before adjusting a setting. If excess concurrency overwhelms a dependency, reducing it may improve sustained behavior even if peak throughput falls. Compare both outcomes rather than declaring either universally better.

Use an isolated trial or a limited canary, with the old value, restoration path, duration and stop conditions recorded. Keep the original workload mix and unrelated deployments out of the comparison.

For the shared connection pool, measure long reports and small queries separately under the same offered load. More total completions cannot justify the setting if interactive requests miss their objective. Count accepted, rejected and completed requests alongside latency; refusing more work can make successful-request latency look artificially good.

Judge a tuning change under the same offered load

Read diagram description

A lower latency value can result from rejecting more requests. Compare acceptance, rejection, completions and latency together before deciding whether the change meets the user-facing objective. Diagram labels: Same offered workload: Keep the request mix and test conditions comparable; Accepted requests: Measure completed work + latency; Rejected requests: Measure the work refused or deferred; Service decision: Does the tradeoff meet the user-facing objective?.

If these same long reports instead point you toward memory growth, the next experiment changes with the explanation. Suppose the process grows while assembling a large report. Watch its memory while the report is built and after the work finishes, keeping report size and concurrency steady. A brief peak during construction and growth that remains across later jobs give you different problems to investigate. The virtual-memory guide describes individual controls; it does not supply a universal low-latency preset. Choose a control only after the observations explain which memory behavior you are trying to change.

Emergency containment may need several coordinated changes. Preserve their sequence and uncertainty, then separate effects afterward. A neat experiment should not delay a necessary mitigation.

Keep the reason with the persistent value

Before persisting a setting through configuration management, compare user outcomes, resource consumption and failure behavior with the baseline. Record the hypothesis, adverse effect and reversal condition so a later owner can reconsider it.

For the shared pool, the conclusion must name both the work that finished faster and the work that waited longer. The filesystem guide and metrics guide help investigate their respective details. Keep that account with the chosen setting, and the next rising throughput graph will have enough context to say whether the service improved with it.