Following one checkout through OpenTelemetry

In this illustrative checkout investigation, you are trying to find out why a customer had to wait. The payment chart looks slow. So does the database chart. A queue is growing nearby. You have three plausible explanations and, so far, no way to tell which records belong to this particular purchase.

OpenTelemetry helps the services leave a connected account of the work. Follow one request into its payment call, read the events recorded along the way, then compare it with other checkouts. The useful change is not another chart. It is being able to follow the same customer request when the work crosses into another service.

One checkout, three different questions

Start with the wait itself. A record of the checkout operation can show when it began and ended. Another records its call to payment. These operation records are called spans; related spans form a trace. Together, they let an investigator follow the request instead of guessing which durations belong together.

Now suppose payment tried and failed. A log records that individual event. Meanwhile, a metric can summarize how often checkout fails across many requests. The trace helps with “What happened here?” The log supplies an event worth examining. The metric helps answer “How many people are having this problem?” One troublesome checkout does not tell you the size of the crowd behind it.

OpenTelemetry supplies the shared software interfaces, development libraries and naming conventions for producing and sending these signals. Its history brings together OpenTracing and OpenCensus, two earlier projects working on this problem. It does not provide the whole storage and investigation system: another tool stores the records and lets you explore them.

The same relationships help when following a document through processing or a request through an internal application. The point of collecting traces, logs and metrics together is that each can answer something the others cannot. For our checkout, connecting the application to payment might settle which service needs attention. That gives the collection work a useful first destination.

With an unfamiliar service, you may need a broader initial collection to discover what matters. That costs more, but looking only for your first suspected cause can hide the actual problem. Put a limit on that exploratory phase and revisit its expense once the recurring questions become clear.

Payment needs to know which checkout sent it

A payment service handles many requests. Being told that it was slow around the same time as checkout is not enough to pick the right one. The calling service needs to send identifying information with the request, so payment can record where its work belongs.

In this simplified teaching example, the checkout trace is T. Its checkout span is C, and its outgoing payment call gets span P. The instrumentation, the software that creates these records, sends T and P with that call. Payment reads them and creates its own span Q, still in trace T, with P as its parent.

The letters are teaching labels, not real wire-format identifiers. A span also records an operation's name, timing and useful descriptive attributes. Here, sharing T says “same request”; recording P as Q's parent says which operation called payment. The records can arrive separately and still be connected by the backend.

Sending that identifying information across the call is called context propagation. OpenTelemetry's propagation guide explains how it travels, and its trace guide shows the resulting relationships. If both services produce spans but lose the context between them, you can have two perfectly good timings and no reliable way to join this checkout to this payment. Conversely, sending context to a payment service that records no span will not conjure one into existence.

Logs can carry the trace and span identities too, bringing payment's failed attempt alongside its operation. Metrics still describe the wider workload. Where supported, an exemplar links a measurement to a sample trace, giving you a way to move from a population-level view into one recorded request.

That connection is useful without explaining everything. A trace can show one operation calling another; it does not establish that a nearby software release caused the delay. Nor can you simply add every displayed duration. The checkout span may already include the payment wait, and some operations overlap. Counting the same wait twice would make the customer seem more patient than they had to be.

The records have a journey of their own

Once checkout and payment have produced their records, those records still need to reach the investigation tool. Applications can send them directly. Another option is the Collector, a separate service that receives telemetry, processes it and exports it onward. Depending on its configuration, it can group records into batches, filter or enrich them, and route them to a backend.

That gives you a useful place to control delivery, and another service to look after. If the backend stops accepting data, what happens to the waiting records? Watch the Collector's queues and memory, refused data, drops and export failures. Making the backend unavailable in a test environment lets you examine how the application libraries and Collector trade lost telemetry against interference with the application.

There is also a difference between information worth keeping and information worth repeating everywhere. Remove sensitive fields before export. A trace identity helps you find one request, but using a unique request ID as a metric label can create a separate series for every request. Cardinality is the number of distinct label combinations: a stable service name groups work, while a unique request label keeps making new groups. That choice can become expensive quickly.

The selection of records matters too. Sampling to catch rare failures serves a different purpose from sampling to estimate how often a population fails. Keep that purpose attached to the resulting view, rather than treating every selected group as an ordinary cross-section of traffic.

For a missing checkout trace, follow a known event through three places: where the application emits it, where it is received, and where it is exported. Each stop narrows the search. An alert about the broken collection path needs a surviving route to a person as well. Sending that alert through the same failed path leaves the investigator waiting for two things that will never arrive.

From the application to the person investigating

Follow the diagram in words

The instrumented application produces traces, metrics and logs with a stable service identity. The application instrumentation creates spans and propagates context across calls; the export library sends the resulting records onward. An optional Collector receives, processes and exports the records, exposing its drops and queues. The backend stores the records and makes them searchable; their freshness still needs checking. The responder uses them to connect a user’s symptom with the recorded work. These are the diagram’s application, SDK export, Collector, backend and responder stages, not a claim that every installation needs a Collector.

Give yourself one slow checkout to find

A useful first trial can be small. In a test environment, choose the question you want the records to answer, perhaps whether checkout waits on payment or spends its time retrying. Give the services stable identities, carry the context across the relevant calls and deliberately make one test request slow. You now have a particular event to look for, rather than an empty dashboard to admire.

Follow that request into the backend. Its trace, related logs and aggregate measurements should tell a consistent story. Then interrupt telemetry in the test environment and check that the missing observations become visible. A slow payment and a lost payment span can look similar from a distance; this trial helps you learn where to tell them apart.

The instrumentation guide takes that work into rollout. Back at our checkout, the chart can now lead to the payment call and its events instead of another round of guessing. And if that call's record is absent, you have somewhere to begin: did the service emit it, did the next stage receive it, and did it reach the tool you are looking at?