Observability for SRE

A customer clicks Pay and waits. Your dashboard agrees that checkout is slow, but agreement is not much help yet. Did the application spend the time calculating the order, waiting for a database, or asking the payment provider for an answer? Observability is what lets you follow that question into the system. Metrics, logs and traces expose different parts of its behavior, so you can investigate a delay instead of merely watching it.

Consider an illustrative checkout service whose latency rises after a release. The release is a useful lead, but several explanations are still possible: a changed application path, a slow dependency or an overloaded resource. The signals become useful when they help distinguish those possibilities.

Which checkout requests became slow?

The first question concerns the affected population. Is latency elevated across all checkout requests, in one region or only for a particular operation? Request volume, error rate and latency distributions establish the extent of the symptom. Successful and failed requests need separate attention because a fast error response can make an aggregate latency figure look better while customers still cannot complete checkout.

The guide to service metrics explains these measurement choices and denominators. Once the affected route and interval are clear, an individual slow request becomes worth examining.

From a slow request to its execution path

A trace follows one request through the operations you have instrumented. Each recorded operation, called a span, has timing and context. If checkout takes three seconds and a payment call occupies most of that interval, the trace gives you somewhere specific to investigate. It has located the wait; it has not yet explained whether the payment service, network or client caused it. The tracing implementation guide follows the instrumentation needed to make that path visible. Sampling and missing spans affect which requests you can inspect and how much of their execution you can explain.

Logs can supply the next piece: perhaps the payment call timed out, or the client made another attempt. Structured logs put details such as the event, service and request identifier in fields you can search. A shared request or trace identifier connects that event to the slow execution, instead of leaving you to guess which of a thousand nearby messages belongs to it. The logging guide discusses what context is useful without collecting unnecessary payloads. Metrics summarize the affected population; traces and logs provide detail about particular executions. Profiles and other evidence may be needed when those signals leave the question unresolved.

Testing the release hypothesis

Now return to the release that first looked suspicious. A deployment timestamp tells you what changed nearby in time. To test whether it matters, compare affected and unaffected requests: the new and old release, the slow region and a healthy region, or the payment route and another operation. If representative traces show extra time in one dependency, compare that dependency's behavior and the affected request cohorts. If the traces omit the slow operation, the instrumentation gap itself explains why the dashboard cannot answer the question. An empty query may reflect lost or delayed collection, so data freshness belongs in the investigation too.

After an intervention, the customer-facing measurements provide the recovery check: successful checkout completion, latency and errors for the affected population. The investigation also needs to account for adverse effects on other paths. Google's monitoring chapter explains latency, traffic, errors and saturation; the OpenTelemetry primer develops the relationship between telemetry and investigation.

How the measurements inform reliability decisions

A service level objective describes the experience the team intends to protect. Its error budget uses the indicator's units. For example, a time-based 99.9% objective over exactly thirty days allows 43.2 minutes of bad time. A request-based objective instead counts eligible requests and bad outcomes; the two allowances answer different questions.

The error-budget policy guide connects that consumption with release decisions, exception authority and recovery work. In a review, the measurements explain what happened to the service; the agreed policy determines what that means for the next release.

When checkout depends on an AI step

An AI-dependent request adds more behavior to explain. Token use, time to first token where relevant, total latency and tool execution can help locate cost or delay. Task outcome still needs its own check: an HTTP success says the request was handled, not that an answer was correct or the requested work finished.

The AI token guide covers cost, latency and context limits, whose accounting and truncation behavior depend on the model and interface. The production-agent guide follows the state and external actions that can extend beyond a model call. Keeping these signals attached to the same user journey makes the AI step investigable with the rest of the service.

Closing the gap the investigation exposed

The investigation ends with a more useful question than “Which monitoring tool should we buy?” You can name the missing evidence: the payment span, the retry event, or the release label that would separate two explanations. That gap determines the next implementation step. Use Python collection and aggregation when inputs cannot be assembled, logging configuration when events lack context, or the OpenTelemetry guide when responders cannot connect a request across services. For container platforms, resource and lifecycle behavior supplies another part of the explanation. With those inputs available, AIOps investigation assistance has something concrete to work with. The customer began with a slow checkout. You now have a way to follow the wait, test an explanation and check whether the experience actually improved.