Will the payment timeout survive your tail-sampling policy?

The cart loads, checkout begins, and everything looks routine until payment times out. That last event changes which records an investigator wants to keep. Tail sampling waits for recorded operations to arrive before choosing which traces to retain, giving a later failure a chance to save the earlier context. The Collector's default trace-complete strategy works this way. Its useful bargain is to keep fewer uneventful requests without losing the ones that explain trouble. The important ending has to arrive before the decision, though.

A good first trial is small enough to follow by hand: reproduce a known checkout failure on one important path, then retrieve it under the proposed policy. Before celebrating a smaller storage bill, make sure the remaining record still explains why payment failed.

Why failures suddenly fill the search results

Consider 1,000 illustrative checkouts: 20 errors and 980 fast successes. Suppose every error arrives in time and is kept, while successes have only a 5% chance of retention. The expected result is 20 errors plus 49 successes, about 69 traces. Errors now make up roughly 29% of stored traces, although only 2% of requests failed. An actual finite sample will vary; these are expected counts, not measured Collector output.

That change is useful for investigation. It is misleading if someone reads the saved traces as a census of traffic. Use full request counts to measure reliability and the retained traces to explain particular cases. First, those cases need to arrive intact enough to be recognized.

Follow the missing payment record upstream

Imagine three recorded operations: checkout, the client calling payment, and the payment server handling it. Each record is a span; their shared trace ID ties them to the same request. When the timeout's recorded error arrives before selection, an error rule can retain the earlier successful operations too. Waiting gives the sampler that later information.

Head sampling makes the keep-or-drop decision earlier, often in the application's instrumentation library, or SDK. It can avoid work, but it cannot see a future failure. Tail sampling sees only the spans that reach it and cannot recover discarded ones. Put it behind a 10% head sample, and it can retain qualifying errors within that sample, not every application error. The sampling overview explains the two approaches.

Suppose checkout appears in the trace but payment does not. Before changing a retention percentage, work backward: did payment produce a span, did the trace ID travel with the call, and did an SDK or a decision inherited from the parent operation discard it? A later rule cannot supply a record that was never sent to the sampler.

Use a synthetic staging transaction without real payment data to follow one trace ID from checkout to payment. Collect the full span set on that path first, so there is a known result to compare with the sampled one. Our implementation guide covers choosing the path. A complete retrieval gives the trial a concrete starting point.

Keep the parts of one request together

Now imagine splitting those records between two samplers. One sees payment fail; the other sees only the earlier successful checkout work. Each can apply its rules correctly while the investigation loses half the story. Routing is part of getting the answer right, not merely spreading the traffic.

A common arrangement receives spans in one tier and selects traces in another, using the load-balancing exporter to route by trace ID. Intake instances need a consistent set of destinations. When that set changes, some IDs move to different samplers. Adding capacity can therefore split work that is already waiting, unlike scaling a service that holds no request state.

One sampler removes that routing variable from the first trial, but it is not a highly available design. Before expanding, use staging to compare retrieved span sets with multiple intake instances and deliberate membership changes. Look before, during and after the change, not only once the new arrangement has settled.

Order transformations around what the rules will actually see. Add needed attributes and remove sensitive data before rules that depend on the resulting fields. A rule cannot match a field that privacy processing has already removed. The gateway guide helps separate those transformations from selection on the deployment map.

Give the timeout two ways to be kept

Our payment timeout could qualify because it records an error or because the recorded trace takes long enough. Fast, successful checkouts are useful too: they show what ordinary work looks like. This first policy retains a small random sample of those. Its percentage is a trial setting, not a measured saving.

The fragment below needs a Collector distribution that includes the processor. It is not a complete production configuration or a benchmark. Merge it into the intended pipeline without losing receivers, authentication, memory protection or exporters, and validate it against the pinned build before a trial.

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 20000
    expected_new_traces_per_sec: 300
    policies:
      - name: checkout-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 1500
      - name: ordinary-comparison
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

The processor reference describes how policies combine. In this fragment, either the error policy or the latency policy can retain a trace independently of the 5% rule. Adding an explicit drop policy would change that logic. The arrival and capacity settings are illustrative as well.

If payment's span arrives in time with an explicit ERROR status, the error policy can keep the trace even when the 5% rule would not. A different checkout that succeeds slowly can qualify through latency. A fast success may be absent on purpose. One successful lookup cannot show that the random sampling rate is right; that takes many distinct trace IDs. These are cases to send and retrieve after decision and export delays, not tests already run for this article.

Change just one detail: the application catches the timeout but never sets a span error. A log containing the word timeout will not make the status policy match. An HTTP failure does not necessarily mark every span as an error, and caught timeouts can differ from uncaught exceptions. The latency rule may still keep the trace, using the received timestamps and its threshold, not the service's entire performance objective. The saved trace ID lets you inspect which parts actually survived.

Thirty seconds of waiting adds up

At a steady 300 new traces per second, a 30-second wait suggests roughly 9,000 traces waiting in memory. A burst of 900 per second for the same interval suggests 27,000. That already exceeds the fragment's 20,000-trace setting, before anyone has established that the process has enough memory.

Our checkout takes one trace slot whether it contains six small spans or hundreds of tool spans. Counting traces therefore cannot establish memory safety. Measure representative span counts and sizes for memory and CPU use, leaving room for export queues and the other processors too.

A longer wait might let the payment error arrive, but every pending trace waits longer with it. Memory use rises and retrieval takes longer. Measure from the first span received to the last span needed for the investigation. Checkout's own duration misses time spent waiting in SDK batches and traveling through transport.

For an hours-long job, stretching the short-checkout arrangement indefinitely is unlikely to be a useful answer. Such work needs its own path or policy with linked stages. A trial that preserves checkout failures has not yet shown that background failures will survive.

Move the error past the decision

After the ordinary checkout can be retrieved, deliberately delay payment's only ERROR span until after selection. The sampler must decide from an incomplete story. Then test an arrival burst, and separately a restart while work is pending. Keeping these experiments distinct makes a missing checkout much easier to explain than changing all three conditions at once.

A decision cache remembers earlier keep-or-drop choices; it cannot make the late error arrive in the past. It can guide later spans according to the remembered choice. What happens after that depends on the pinned version's treatment of late spans and traces removed early. Component measurements help locate a loss, while the known checkout ID connects those numbers to the request the responder was looking for.

Change one condition	Follow the result	Decision it informs
Let payment's error arrive on time	Retrieve checkout and payment spans with the saved trace ID	Whether the normal path preserves the intended investigation
Delay the error beyond the window	Compare the initial choice, late-span handling and final trace	Whether to adjust the wait or separate this workload
Send an arrival burst	Check known traces, memory, refusals and early removals	Whether capacity holds beyond average traffic
Restart a sampler or change membership	Compare missing span sets and recovery time	What loss to expect during maintenance

A retained trace still has to leave the Collector. Export queues protect different state from the spans waiting for a sampling decision, and export can fail independently. Follow the resiliency guide through that part of the path, then perform the final backend lookup a responder would use.

For checkout, the result needs enough detail to distinguish time spent before payment from the payment timeout itself. A record saying only that checkout was slow has preserved the symptom, not the explanation. That is the difference the routing, waiting and export trial is meant to expose.

The discarded checkout still made part of the trip

A fast checkout rejected by the sampler has already sent its spans from the application to that sampler. Reducing later retention does not undo that traffic. Compare retained bytes and backend ingestion with Collector operation and maintenance, using actual billing and compression. Keeping 5% of ordinary traces does not remove 95% of the entire bill.

Include the investigator's wait as well. Can they find the request, follow its work and explain the timeout from the sampled result? How long does retrieval take? Missing spans during bursts or restarts belong in that comparison, rather than being filed away as an unrelated inconvenience.

Remember our 1,000 illustrative checkouts: errors became about 29% of stored traces despite a 2% request error rate. That concentration helps someone find failures but cannot describe fleet reliability. Population metrics provide the full counts underneath the rates; selected traces explain executions. Our trace-reading guide explores what missing and overlapping observations let you conclude.

Privacy follows the trip too. A payment field discarded at the end has already passed through the application, transport and sampler. Keep sensitive material out at instrumentation or another appropriate early step, restrict access, and secure transport across trust boundaries. The sensitive-data guidance explains why saving fewer traces is not, by itself, a privacy measure.

Keep a route back that the backend can handle

Tail sampling is worth operating when the retained timeout remains useful and the savings justify the measured work of running it. At low volume, keeping everything may be simpler. If payment's span cannot reach the same sampler in time, changing a percentage will not repair the trace. The path or design needs attention.

Save the full-retention configuration, but rehearse returning to it within the pilot's scope. It sends more traffic to the backend, which needs enough ingestion capacity for that return. Otherwise the recovery can lose records again. Restoring one known path is a smaller change to understand than switching an entire fleet during an incident.

Come back to the checkout that started the trial. If the limited set still lets a responder follow the work and explain the payment timeout, the saving has kept what mattered. If that request disappears, the lower ingestion meter is measuring a cheaper system that cannot answer the question.