Opens in a new tab
Guide

What an AIOps assistant can add to an investigation

In brief

A slow order, a release and a dependency outage show where AI-assisted observability helps, and what the responder still needs to establish.

6 min read

Is this the right starting point?
Before you read
Basic service telemetry, deployments and incident-investigation concepts; use authorized representative records for comparison.
What you will take away
Evaluate AI-assisted investigation by source-supported comparisons, missed evidence, review effort and coverage rather than fluent causal summaries.
Sources
Blue and orange vans both queue at one crowded bridge while both colors move on an alternate open route.
On this page5 sections

An order slows down after a release. A dependency has failed in the same window. An AIOps assistant finds both records and puts them in front of the responder, who now has a useful question to pursue. Did the release change the order's behavior, or did the order wait on the failed dependency?

This hypothetical investigation gives AI-assisted observability a concrete job. Help the responder find the evidence and comparisons that separate those possibilities. A confident paragraph about the release would be quicker to read, but blaming the release too soon could steer the team toward the wrong mitigation.

Follow the order through the services it used. Which orders slowed, what path did they take and what changed along that path? The assistant can help assemble the trail. The important part is whether the trail lets the responder make a better-supported choice.

Finding a pattern and explaining it are different steps

An anomaly detector looks for behavior that differs from what it expects, such as unusual order latency. That can tell a responder where to look. Unusual behavior still needs interpretation, since a harmless seasonal change and a customer-impacting fault can both depart from an earlier pattern.

Correlation can bring related-looking events into one view, including our release and dependency outage. Their shared timing gives the investigation a starting point. Two failures can occur together without sharing a cause, so the responder still needs a way to compare them.

A language model can turn the records into a short account and suggest that comparison. Suppose its summary says the release caused the slowdown but leaves out the dependency outage. The prose has become easier to read while the explanation has lost an important possibility. Keeping the original records accessible gives the responder a chance to catch the omission.

Finding evidence and proposing a useful investigation are valuable roles in their own right. Suppressing a page or changing the service introduces different consequences. Evaluate those powers separately from the quality of the assistant's analysis.

Separate the requests before choosing a cause

For an illustrative comparison, hold the region, order operation and observation window constant. Compare requests handled by the old and new application versions on the same dependency route. If only the new version slows, that supports further investigation of a release regression.

Now compare routes. If both application versions slow when they use the affected dependency, while requests on an unaffected route do not, dependency trouble gains support. Neither comparison establishes root cause alone. Missing version or route records may leave the choice unresolved.

An assistant that suggests this split can advance the investigation before the cause is settled. That does not give it permission to turn the provisional explanation into a rollback. Keep the proposed check, what it found and the remaining uncertainty together.

Measure the responder's active review effort alongside elapsed time. A short answer may take a long time to verify. Record harmful omissions separately from wording defects, because a missing dependency and an awkward sentence require different corrections and carry different consequences.

Compare an assistant’s hypothesis with the request evidence
Nearby deployment and symptom records suggest a hypothesis. Compare requests, dependencies or unaffected traffic, then support, reject or leave the hypothesis unresolved.
Illustrative investigation. An assistant can connect a deployment and symptoms, but the responder still needs evidence that distinguishes that explanation from alternatives.
Read diagram description

Illustrative investigation. An assistant can connect a deployment and symptoms, but the responder still needs evidence that distinguishes that explanation from alternatives. Diagram labels: Deployment + symptoms: Events close together in time; Assistant hypothesis: Possible relationship with linked evidence; Discriminating check: Compare requests, dependencies or unaffected traffic; Investigative result: Support, reject or leave the hypothesis unresolved.

Silence has several possible explanations

The assistant reports nothing useful. Did fresh records arrive? Did collection lose any? Did the analysis run, and did its answer reach the responder? An empty finding only becomes meaningful when the team knows the system had evidence to examine.

Keep dependable monitoring for known customer-impacting failures while evaluating the new capability. If suppression or remediation is proposed, give that change its own assessment, owner and stopping condition. A trial of better analysis should not quietly change the existing failure coverage.

The alert-correlation guide and anomaly-detection guide examine those failure modes in more detail. Here the response needs both a route back to the original records and an account of whether collection was complete.

Keep the answer attached to the order's records

Service identity, event time and deployment context place the signals in a shared investigation. Where appropriate, request or trace identifiers connect the overall slowdown to individual orders and the services they used. The observability guide for SRE develops that route from symptom to mechanism.

Suppose the assistant says only the new version slowed, but its cited old-version records contain no dependency-route field. We cannot yet tell whether those requests used the affected route. Keep the missing field visible, compare the deployment records with the current incident state and look for the route evidence. The summary has raised a check that the records cannot yet complete.

Record that choice and any later recovery evidence so the next responder can see why the team left the cause unresolved. The OpenTelemetry primer explains the telemetry and observability background. Instrumentation supplies evidence; the investigation must still determine what that evidence supports.

Count useful leads and the work they require

Compare the current process and the candidate assistant on a held-out set of representative incidents and healthy periods, with the same evidence available to both. Include incomplete telemetry, a harmless surge, simultaneous failures and a release unrelated to the outage. Those cases ask whether the assistant can retain distinctions that a neat single-cause account might erase.

Imagine that reviewers evaluate twenty cases and receive useful, supported leads in sixteen. That is 80 percent of this illustrative sample. It says nothing by itself about an 80 percent reduction in incident duration. The other four could include an unsupported lead, a missed dependency, stale telemetry and no output, each pointing toward a different repair.

A provisional lead can be useful when it is clearly labelled and inexpensive to check. A consequential service change needs stronger support because testing an idea by opening its source records has a different cost from testing it by changing production.

For the missed dependency, return to the original order records. Did collection omit the event, or did the assistant receive it and leave it out? That determines whether the repair belongs in collection or answer generation. Time the work needed to find and correct the omission, including contradictory records the responder had to inspect.

Include consequences the trial actually measures, such as unnecessary pages, harmful actions or the point at which recovery was verified. Preserve missed incidents and incomplete-data cases too. Sixteen useful answers do not erase the other work, and four unsuccessful ones need not share a cause.

Back at the slow order, the responder needs to open the sources, separate the request groups and explain what would support either possibility. An assistant earns a useful place when it shortens that path while keeping the unanswered question visible.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next