Opens in a new tab
Home/Observability for SRE

Observability for SRE

A customer clicks Pay and waits. Your dashboard agrees that checkout is slow, but agreement is not much help yet. Did the application spend the time calculating the order, waiting for a database, or asking the payment provider for an answer? Observability is what lets you follow that question into the system. Metrics, logs and traces expose different parts of its behavior, so you can investigate a delay instead of merely watching it.

Consider an illustrative checkout service whose latency rises after a release. The release is a useful lead, but several explanations are still possible: a changed application path, a slow dependency or an overloaded resource. The signals become useful when they help distinguish those possibilities.

Which checkout requests became slow?

The first question concerns the affected population. Is latency elevated across all checkout requests, in one region or only for a particular operation? Request volume, error rate and latency distributions establish the extent of the symptom. Successful and failed requests need separate attention because a fast error response can make an aggregate latency figure look better while customers still cannot complete checkout.

The guide to service metrics explains these measurement choices and denominators. Once the affected route and interval are clear, an individual slow request becomes worth examining.

From a slow request to its execution path

A trace follows one request through the operations you have instrumented. Each recorded operation, called a span, has timing and context. If checkout takes three seconds and a payment call occupies most of that interval, the trace gives you somewhere specific to investigate. It has located the wait; it has not yet explained whether the payment service, network or client caused it. The tracing implementation guide follows the instrumentation needed to make that path visible. Sampling and missing spans affect which requests you can inspect and how much of their execution you can explain.

Logs can supply the next piece: perhaps the payment call timed out, or the client made another attempt. Structured logs put details such as the event, service and request identifier in fields you can search. A shared request or trace identifier connects that event to the slow execution, instead of leaving you to guess which of a thousand nearby messages belongs to it. The logging guide discusses what context is useful without collecting unnecessary payloads. Metrics summarize the affected population; traces and logs provide detail about particular executions. Profiles and other evidence may be needed when those signals leave the question unresolved.

Testing the release hypothesis

Now return to the release that first looked suspicious. A deployment timestamp tells you what changed nearby in time. To test whether it matters, compare affected and unaffected requests: the new and old release, the slow region and a healthy region, or the payment route and another operation. If representative traces show extra time in one dependency, compare that dependency's behavior and the affected request cohorts. If the traces omit the slow operation, the instrumentation gap itself explains why the dashboard cannot answer the question. An empty query may reflect lost or delayed collection, so data freshness belongs in the investigation too.

After an intervention, the customer-facing measurements provide the recovery check: successful checkout completion, latency and errors for the affected population. The investigation also needs to account for adverse effects on other paths. Google's monitoring chapter explains latency, traffic, errors and saturation; the OpenTelemetry primer develops the relationship between telemetry and investigation.

How the measurements inform reliability decisions

Once checkout recovers, the team still has a decision to make: what does this interruption mean for the next release? First they need an agreed description of the experience they intend to protect, a service level objective. The measurements then show how close the service came to that target.

Suppose the illustrative checkout service has a time-based objective: it should meet the team's definition of good service for 99.9% of exactly thirty days. Thirty days contain 43,200 minutes. The remaining 0.1% is 43.2 minutes of bad time. That allowance is the error budget. It gives the team a way to account for interruptions against its target, rather than treating every bad minute as an isolated event.

Now imagine counting checkout requests instead of minutes. A request-based objective counts the eligible requests and how many had a bad outcome. A quiet minute and a busy minute can contain very different numbers of attempts, so an allowance measured in minutes cannot simply stand in for one measured in requests. The budget has to use the same units as the measurement behind the objective.

The error-budget policy guide connects the amount used with release decisions, who may authorize an exception and recovery work. The checkout measurements tell the team what happened. Its agreed policy determines what that means for the release waiting to go out.

When checkout depends on an AI step

Extend the illustrative checkout with an assistant that helps a customer understand a delivery choice. The customer asks a question and waits for an answer. The application may be waiting for the model to start responding, for it to finish, or for a separate tool to return information the answer needs. Those waits belong to the same customer journey, but they point to different work inside it.

A model handles text in pieces called tokens. Where the response is delivered a piece at a time, time to first token measures the wait before the first generated piece arrives. Total latency measures the wait for the whole result. The opening words can appear promptly while the customer still waits for the useful part. If the assistant calls another service through a tool, recording that tool's execution helps locate the additional delay. Counting token use adds another view of the work and its cost.

Then read the answer and check the requested result. A successful HTTP response, the application's response to the web request, says that request was handled. It does not establish that the delivery answer was correct or that any requested work finished.

The AI token guide follows cost, latency and the amount of material a model can work with at once, its context limit. How tokens are counted and what happens when input is shortened to fit that limit depend on the model and interface. The production-agent guide follows stored state and external actions that can outlast a model call. Keep those steps connected to the customer's request, and an unexplained wait in the assistant becomes something you can follow through the service.

Closing the gap the investigation exposed

The investigation ends with a more useful question than “Which monitoring tool should we buy?” You can name what was missing: the payment span, the retry event, or the release label that would separate two explanations. That gap determines the next implementation step.

If the payment wait was invisible, start by making that operation visible. If the event was recorded but nobody could connect it to checkout, repair the connection. These guides cover the different jobs:

The customer began with a slow checkout. You now have a way to follow the wait, test an explanation and check whether the experience actually improved.