Opens in a new tab
Linux basics · Advanced lesson

Why more Linux workers can leave a small query waiting

In brief

A shared connection pool shows how a tuning change can finish more work while slowing some requests. Follow the wait, test the limit and compare both outcomes.

7 min read

IntermediateLessonAbout reading levels
Is this the right starting point?
Before you read
Basic Linux shell, process and service-metrics familiarity; use authorized read access and an isolated environment for tuning trials.
What you will take away
Investigate a resource wait and evaluate a reversible tuning change under comparable offered load and user outcomes.
Sources
Large orange balls and small blue balls lie in two chutes and an amber funnel, with a loose green ball on the tabletop.
On this page5 sections

The reports are finishing faster. That is the good news. Someone trying to run a small query has been watching it wait, and would quite reasonably like to know whether the improvement includes them.

In this hypothetical service, long reports and small queries share a pool of connections. Adding report workers lets more reports hold those connections at once. More work finishes each minute, but the small queries spend longer waiting for a turn. The overall count has hidden a cost paid by a particular group.

Useful Linux tuning begins with that person's wait. Find whether the request needs processor time, storage, a connection or something else, then choose a setting that tests your explanation. The result has to account for both kinds of work. A larger throughput number, the amount completed in a given time, cannot do that alone.

Keep the small queries in the picture

Start by comparing how long the small queries take before and after the extra report workers arrive. Look across their response times, including the slow ones, rather than folding them into the reports' result. Keep completed work and errors alongside them. Record how much of the incoming work consists of long reports and how much consists of short queries.

That mix is easy to lose when comparing two afternoons. If the second afternoon brings larger reports or more requests, the workload has changed along with the setting. You need a healthy comparison interval, or baseline, that lets you see what the setting contributed.

Keep the machine and its limits with that account: kernel version, machine type, container limits and recent changes. A value that helped on one machine inside one quota may behave differently when either changes. These details save the next engineer from repeating an experiment whose conditions have quietly disappeared.

While the query is waiting, the following read-only tools can help you look at the host. They require the relevant local packages; available fields and behavior vary, so consult the installed manuals:

uptime
vmstat 1 5
iostat -xz 1 5
free -m
ss -s

The clock matters as much as the column. The first vmstat report includes averages since boot, though its process and memory readings are instantaneous. The first iostat report normally covers time since boot too. Later interval reports are a better match for the short period you are investigating.

A high load average is not a CPU percentage. Nor does a small amount of completely free memory tell you that the machine has run out: some cached data can be reclaimed, so inspect available memory as well. Each reading helps narrow the question. None can tell you, on its own, which connection the query is waiting to acquire.

Brendan Gregg's USE-method checklist organizes these questions around how much of a resource is in use, whether work is waiting for it, and whether it has errors: utilization, saturation and errors. It gives you somewhere sensible to look when you have a wait to explain.

The processor can be idle while the query waits

Suppose there is still idle CPU time on the host. That fits our shared-pool explanation: a query waiting for a connection held by a report need not keep a processor busy. It also fits other explanations, including a limit on the container. We have narrowed the possibilities, not selected a winner.

The vmstat manual gives the short column names their meaning. r counts runnable processes, either running or waiting for processor time. b counts processes blocked waiting for input or output, abbreviated I/O. id reports idle CPU time. If runnable work stays high while idle time stays low, demand for processor time and its scheduling deserve a closer look. That still does not identify a particular function to optimize.

Return to the delayed query and follow its recorded steps, or trace. How much of its time passes before it gets a connection? If blocked work instead leads you toward I/O, examine device delays and limits. Storage being involved does not yet explain whether the wait comes from a queue, a limit or the way the application accesses it. Choose the next observation for the remaining question, rather than collecting another screen of numbers because it is available.

How much of this machine can the service use?

The host's spare processor capacity may look inviting. The service may not be allowed to use it. Linux control groups, usually called cgroups, group processes and can limit the resources they receive. Check the waiting workload's quota before deciding that the host's idle time rules out a processor limit. Changing a host-wide value will not remove a narrower limit still imposed on that workload.

Open files have a similar distinction: a process can use up its own allowance while the host remains below its larger limit. Inspect the process that is doing the work. The cgroup v2 guide describes the version's interfaces; do not copy v1 paths into a v2 system without checking them. On a managed platform, make any eventual change through the supported workload configuration rather than behind the system that manages it.

For storage waits, keep following the operation. A volume limit, a queue and an access pattern suggest different remedies. Even a familiar storage scheduler may be unavailable or unsuitable on the current kernel and device. Our long reports might hold connections while waiting on storage, but that connection needs to appear in the observations before a scheduler change becomes a useful experiment.

Try less work at once, and count what happens

Perhaps too many simultaneous reports are overwhelming a dependency. Before changing the number allowed to run at once, write down the expected result. Reducing that concurrency may improve sustained behavior while lowering peak throughput. Both outcomes belong in the decision; neither is automatically the better choice.

Use an isolated trial or a limited canary, a change applied to a small portion of the workload. Keep the old value, a way to restore it, the trial's duration and its stop conditions. Hold the request mix comparable and keep unrelated deployments out of the comparison.

For our shared pool, offer the same load and measure reports and small queries separately. If the small queries still miss their response-time objective, more total completions do not justify the change. Count requests accepted, rejected and completed alongside their response times. Turning away more requests can make the successful ones look wonderfully quick while leaving more people without an answer.

Keep the refused requests in the comparison
The same offered workload branches into accepted requests, measured by completed work and latency, and rejected requests, measured as refused or deferred work. Both branches inform the service decision.
A quicker answer among the requests you accepted can hide more work being refused. Compare both branches under the same offered load before judging the setting.
Follow both branches of the comparison

This conceptual comparison counts work that did not receive an answer as well as work that did. It is not a measured tuning result. Diagram labels: Same offered workload: Keep the request mix and test conditions comparable; Accepted requests: Measure completed work + latency; Rejected requests: Measure the work refused or deferred; Service decision: Does the tradeoff meet the user-facing objective?

The reports might lead you toward memory instead. Suppose memory grows as a large report is assembled. Watch it during construction and after the report finishes, keeping report size and the number running at once steady. A brief peak and growth that remains across later jobs give you different problems to investigate. The virtual-memory guide explains individual controls, not a universal preset for fast replies. The observation should tell you which behavior a proposed control is meant to change.

During an emergency, several coordinated changes may be needed to contain harm. Do not postpone necessary mitigation for a beautifully isolated experiment. Preserve the sequence and what remains uncertain, then separate the effects afterward.

Leave the next owner more than a number

Before making a value persistent through configuration management, compare user outcomes, resource consumption and failure behavior with the baseline. Keep the explanation you tested, any adverse effect and the condition that would make you reverse the change. A later owner needs those reasons when the machine or request mix changes.

The filesystem guide can help follow a storage path; the metrics guide can help with what the measurements count. Here, the useful conclusion has to bring us back to the person running the small query. Say which reports finished sooner, how long that query waited and what was refused. Then the chosen setting describes an understood compromise, rather than a good afternoon for one graph.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next