Capacity testing: what happens when one worker stops?

The missing worker is back. Both workers are busy again. For the customer whose report is still waiting, this is encouraging news with one small problem: the report is still waiting.

Restoring capacity does not erase the work that piled up while it was missing. The service has to clear that backlog while new requests continue to arrive. Recovery depends on the capacity left over after serving those arrivals, not on the total capacity shown on a dashboard.

We can see the difference on paper. Give the fictional Dispatch service two workers, each able to complete twelve comparable exports per second. Eighteen exports arrive every second. These are synthetic inputs with equally sized jobs, not benchmark measurements; no infrastructure changes are needed to explore them.

Six reports join the line every second

With both workers running, Dispatch can finish twenty-four exports per second. Eighteen arrive, leaving room for six more. Lose one worker and capacity drops to twelve. Now eighteen arrive but only twelve can finish: six additional reports join the queue each second.

After five minutes (three hundred seconds), that difference adds 1,800 exports to the queue. When the worker returns, capacity rises to twenty-four again. Eighteen new arrivals still need service, leaving six completions per second to clear the backlog. At that rate, those 1,800 exports take another five minutes to drain.

The tidy arithmetic depends on the tidy assumptions. Larger jobs or extra retries could make recovery take longer.

The worker returns; the line bends

Sketch time across a page and queued exports up the side. During the worker loss, the line climbs steadily. At restoration, it turns downward. It does not drop straight to zero. The worker has returned; the accumulated work has not vanished.

It is tempting to divide 1,800 by twenty-four and predict a seventy-five-second recovery. That gives every available completion to old work and forgets the eighteen new arrivals each second. The capacity available to drain the backlog is twenty-four minus eighteen: six per second. Divide by six and we are back to five minutes.

Now suppose arrivals rise to twenty-four per second just as the worker returns. Arrivals now match completions, so the total backlog stays constant. That does not mean the same reports stay in it: if jobs are processed in arrival order, earlier reports can finish while new ones take their places. Both workers can look healthy while a persistent line of work remains. Queue age and completed user operations tell us something the worker status cannot.

Did the report arrive before the promise expired?

None of those throughput calculations guarantees a five-minute completion objective for each export. Time spent waiting, differences in job size and slow dependencies affect individual reports. A real test must observe accepted exports, completed exports, the oldest queue age, missed deadlines and correct downloadable reports.

A CPU chart can help explain a slowdown. It cannot tell the whole story of the customer's request. Measure that operation from a place that can notice its failure, including a report that never becomes available at all.

Choose the stopping point before removing the worker

The paper version is already useful: calculate the timeline and list what a real run would need to observe. If you plan an actual experiment, use a disposable or explicitly approved test environment. Set the workload cap, duration, fault, person authorized to stop it and recovery action before beginning.

One possible experiment removes a worker from test traffic while leaving the dependency in place. Stop at the agreed backlog cap, if resource pressure threatens neighboring tests or if telemetry becomes unavailable. Once you cannot observe the run, adding more load cannot answer the question you started with.

Four thousand reports would be an interesting answer

Our model predicts an extra 1,800 queued exports. Suppose a test instead shows 4,000. Keep that result. Compare the predicted and observed arrivals, completions, peak backlog and drain time. Uneven jobs, retries multiplying the work or a dependency limiting both workers could explain the difference.

Revise the assumption that the observations challenge. Let the environment return to its starting state, then rerun the bounded test. Adjusting the chart until it resembles the estimate would remove the part that taught us something.

Recovery ends farther downstream than the worker

Losing one worker does not test a lost region, corrupted data or an unavailable queue. Keep the scenario, load distribution and duration attached to the result. Label paper predictions separately from observations so the next reader knows which kind of result they are looking at.

When the worker came back, how much capacity went to new arrivals and how much to old work? Did the waiting users then receive their reports? The green worker indicator marks a useful moment. Those answers tell us what happened after it.

Sources

Managing load and large-system design discuss capacity reasoning. Dispatch's numbers and failure scenario are illustrative, not measured results.