Follow one slow status request through the evidence

The dashboard says requests are completing. The customer says the export page is slow. Both can be right. The chart might cover different requests, hide a smaller affected group or stop timing before the person’s wait ends.

To follow the complaint, the system must expose useful information about its work. Metrics summarize measurements, logs record events and traces connect recorded operations. Using those outputs to investigate behavior is observability. Collecting all three is not the same as knowing what any of them describes.

A metric can show how often requests were slow. A log can identify one of them, and its trace can show where the recorded work spent time. The connection matters: each view should help follow the same work rather than start a different investigation.

This paper exercise supplies counts, a log event and a trace. They are enough to identify a slow operation and choose a useful next check. No platform installation or complete diagnosis is needed to ask that next question.

Follow tenant A’s status request

In fictional Dispatch, an export request starts background report work. The user can then poll, repeatedly asking whether it is ready. This complaint concerns slow status-page responses. Report-generation speed remains a separate, unanswered question.

Ask, “During 15:00 to 15:05 UTC, were status responses slow for tenant A, and which part of a slow request should we inspect next?” A tenant is the grouping for an account or customer. Naming A keeps that experience visible beside everyone else’s.

Now the question identifies an operation, a five-minute interval, a group and a decision to advance. “Why is Dispatch slow?” leaves all four open. The customer’s detail gives us something the packet can answer.

A correct fleet percentage can hide a worse wait

The first synthetic observations come from the gateway, the entry point passing requests to the application. Assume it recorded all 1,200 eligible status requests in this interval. Every row defines slow as at least two seconds at the gateway. That threshold belongs to the exercise, not to every service.

For A, 100 of 200 responses were slow: 50%. Other tenants together had 200 of 1,000, or 20%. Across everyone, 300 of 1,200 gives 25%. The fleet figure is correct, but it softens A’s experience and cannot describe each other tenant separately.

These are requests, not people. One person might send several. Half the requests therefore does not mean half the people, nor does it tell us how many minutes the service was slow. It tells us how often this measured operation crossed the threshold among these requests.

The gateway clock excludes browser work and requests that never arrived there. Complete gateway coverage is not complete coverage of a person’s experience. In a real investigation, missing measurements would need attention before trusting the denominator.

The counts suggest looking more closely at A. Choose one identifiable slow request, but keep the population result alongside it. We are moving from a pattern to a particular case, not replacing one with the other.

Find one request, then inspect its timeline

This illustrative event identifies one request from A. It contains no real customer identifier or payload:

2026-10-19T15:02:10Z
service=dispatch-api
operation=export-status
request_id=example-request-17
trace_id=example-trace-17
tenant=tenant-a
status_code=200
duration_ms=2400

The request and trace labels give us a way to follow it. HTTP 200 means the API returned a successful response. It does not make 2,400 milliseconds quick, and it does not show that the background export finished.

These readable labels are for the paper exercise. Real instrumentation must use its tracing system’s required identifier format; this is not an OpenTelemetry wire-format example.

The matching trace groups timed operation records called spans. Software must be instrumented, or configured to report those operations, for them to appear. A child names its parent operation, though child timing is not always contained inside the parent. This simplified trace contains two records:

The child begins at 1,800 milliseconds and lasts 400, ending at 2,200 inside the 2,400-millisecond request. It accounts for part of the wait. The first 1,800 milliseconds and final 200 remain unexplained by the trace.

That blank time is useful: it marks where our account stops. It might contain waiting, local computation or uninstrumented work. And remember which operation this is. A request asking for status need not contain the background export itself.

One next check could time the interval between entering the status handler and gaining permission to read state. Perhaps a concurrency limit makes it wait for a permit. If that wait occupies the gap, inspect the limit and the work holding permits. If permission is immediate, look elsewhere in the handler. Neither timing is supplied here. Calling the whole 2,400 milliseconds a slow storage read would ignore the child’s actual 400 milliseconds.

One slow trace is still one case

The shared label and matching duration connect this log with this trace. They let us inspect one request, not establish how often others took the same path or why A’s slow-response fraction was higher.

If someone chose this trace because it was slow, it is an interesting case rather than a representative sample. Use the table for the population fraction. One selected trace cannot replace those counts. Sampling and collection costs come later; this distinction matters already.

We now have the recorded frequency of a symptom, an event identifying one case and an interval that case leaves unexplained. Together they narrow the next investigation without announcing a cause.

Leave the next investigator a precise question

Put that account into four short entries:

Observed symptom and population:
Evidence supporting that statement:
Explanation not yet established:
Next check and what it would distinguish:

A worked note could say that 50% of A’s 200 recorded status requests took at least two seconds at the gateway in this window, citing the table. The selected request lasted 2,400 milliseconds, of which 400 belong to the recorded child. The rest remains unexplained by this trace.

Leave report-generation speed and cause unresolved. The next check belongs in the status handler’s pre-child interval, not a worker restart for different work. If later observations justify another check, name the question it would answer.

Keep the note with the packet. The apparently healthy chart and slow page can now coexist: they may describe different populations or parts of a wait. The selected trace also has a specific gap worth investigating. The next lesson maps the request and background job so the next measurement can fill a known gap.

Sources and example scope

The OpenTelemetry observability primer introduces observability, metrics, logs and traces. OpenTelemetry’s trace documentation explains spans and their relationships; Google’s monitoring chapter discusses user symptoms and system observations. The site’s trace-reading guide develops concurrency, sampling and unexplained intervals. All counts, identifiers and timings are synthetic and were checked for internal consistency during original preparation. No telemetry platform or production service was exercised.