Cloudflare Traces: following the wait into the origin

Imagine asking an incident assistant for a runbook and watching the answer arrive late. The request got through Cloudflare quickly. Somewhere beyond that point, time went into reaching the origin, retrieving the runbook or waiting for the model. Knowing that the origin was slow is a start. Knowing what it was doing is much more useful.

Cloudflare Traces records actual request processing through supported platform operations and can connect it to instrumented application spans. A span is a record of one operation’s timing and related information. Passing along a trace identifier connects records; it does not create measurements inside the application. Our runbook search is hypothetical, with no deployment or performance result reported here.

Follow the request past “waiting for origin”

The assistant sends its question to /ask. That route calls the origin, which retrieves a runbook and then requests the model’s response. A view ending at “waiting for origin” encloses both jobs in the same unexplained interval. It cannot yet tell us whether to improve search or investigate the model call.

Start with a request you can find again and the question you want it to answer. The first-path tracing guide takes this approach before broadening coverage. For this request, retrieval and the model call need separate records within the same journey. More unrelated traces would not fill that particular gap.

Parent and child spans connect an operation to work performed within it. Cloudflare’s overview describes its hierarchy of supported platform spans and lookup by Ray ID, the identifier associated with the Cloudflare request. Do not confuse that recorded journey with the separate Cloudflare Trace configuration simulator. The simulator predicts rule behavior rather than showing a request that actually ran.

The domain and Worker each need tracing

Enable tracing for the domain receiving the test request. In the domain configuration, persistence keeps traces available in Cloudflare’s dashboard, while export sends them to a configured OpenTelemetry destination, a collector for this measurement data. The runbook investigation needs that exported platform record where it can be joined with the origin’s own spans.

The Worker has a separate setup. The Workers tracing reference describes its automatic handler, fetch, binding and RPC instrumentation. Put this fragment inside the test Worker’s existing Wrangler configuration; it does not replace the application:

{
  "observability": {
    "traces": {
      "enabled": true,
      "head_sampling_rate": 0.05
    }
  }
}

Here 0.05 is a fraction: five percent. The domain field uses a percentage instead. Copying the same number into both would select different proportions of requests. Five percent is an illustrative allowance for this example, not production advice. Enable Workers tracing without an explicit rate and every invocation is traced by default.

Timing a runbook lookup does not require recording the runbook’s contents. Inspect the collected attributes before extending the test, and keep prompts, credentials and confidential text out of custom attributes. Review the external destination’s ingestion and retention arrangements too. Joining domain traces and Workers events does not give them a shared price or retention window.

An identifier connects measurements; it does not make them

Forwarding W3C trace context in the domain settings lets an instrumented origin attach its spans to the request. Context carries the identifiers needed to relate the work, not the timing records themselves. Instrument retrieval and inference at the origin, then send those spans and Cloudflare’s spans to the same OpenTelemetry destination to inspect them together.

Accepting incoming context is another setting. Cloudflare documents Reject as the default and warns that accepted context is not verified as coming from a trusted caller. Two records sharing an identifier can belong to the same trail without proving who supplied it. Keep acceptance consistent with the application’s trust model; the identifier is not authentication.

Open the slow request and look for retrieval, followed by inference. If the view shows only the fetch to the origin, it has located the wait from Cloudflare’s side without explaining the work inside. Capture the missing operations before choosing a component to tune.

Keep the test request long enough to find it

Suppose an authorized test request becomes slow, but its trace was omitted by sampling when it began. The later delay cannot bring that record back. Keep the request identifier, time and expected steps so that “we sent it” does not turn into “we recorded its journey.” This selection at the start is called head sampling.

For a bounded reproduction, a domain rule can retain 100 percent of requests on a restricted test path while the experiment runs. It must be the first matching rule: Cloudflare uses that match, not the rule the operator had in mind. Set Worker and origin sampling explicitly too, or the platform record may survive while the retrieval span disappears. Restrict the input as well as the route, then restore the agreed settings.

In that proposed experiment, deliberately delay runbook retrieval at the test origin. The expected trace has a longer origin-facing interval and a retrieval child span showing where the wait occurred. Find the request by Ray ID and open its corresponding application span at the shared destination. A missing child, failed export or broken parent relationship is a collection problem, not proof of fast retrieval. This is a test plan, not a measured waterfall.

A streaming answer can begin before the bar ends

The responder might already be reading the first words while the platform bar continues. The span reference explains that dynamic includes connection setup, request sending, waiting and transfer of the complete response. The response span includes delivery to the client. Neither ends simply because a streamed answer has started.

A long origin-facing interval alongside a long retrieval child would support investigating retrieval in this hypothetical test. Without the child, several explanations remain. Do not add every bar: parents include work shown in their children, and concurrent operations overlap. The guide to reading distributed traces explains how to read those relationships.

If retrieval never appears, blaming inference would fill the empty space with a guess. Capturing the missing operation is the next useful change, not swapping the model. When the deliberately delayed retrieval does appear, the engineer has a specific path to examine.

The trial has answered its question when another engineer can follow that request from the edge to the slowed retrieval. Its identifier and collection settings make the path recoverable. Repeating under the intended operating sampling policy then checks what will remain available outside the bounded test. The long wait has become work someone can investigate.