Opens in a new tab
Guide

Ask your incident assistant for the next useful check

In brief

In a hypothetical incident, errors began at 14:02 and the rollout started at 14:05. A reusable prompt helps an assistant turn that mismatch into a useful next check.

7 min read

Is this the right starting point?
Before you read
Basic incident-response and AI-assistant familiarity; no model-training knowledge is required.
What you will take away
Draft and evaluate evidence-grounded operational prompts that distinguish observations, hypotheses, useful next checks and enforced permissions.
Sources
A cream rectangular frame surrounds a coral starburst on the left and a mustard-yellow flag on the right, both over a horizontal dark-blue strip.
On this page4 sections

Imagine a responder comparing two lines in an incident record. The errors begin at 14:02 UTC. The recorded rollout, when the new version began reaching the service, starts at 14:05. It is an awkward three minutes for an explanation that blames the rollout for starting everything.

This is a hypothetical investigation, and it is still very much open. An earlier change may be missing from the record. The 14:05 rollout may have made things worse. But an assistant asked only “What caused this incident?” has been given a much bigger job than those two lines can support.

A more useful request is to help choose the next read-only check: something the responder can inspect without changing the system, whose result would separate the explanations still in play. Give the assistant the observations and ask what each one supports, what remains uncertain and where to look next. That can move the work forward without needing a final cause to put at the top of the answer.

Put the two timestamps where the assistant can use them

The assistant needs more than the sentence “errors after deployment.” Supply the service, the start and end of the period being examined, its time zone, and the measured impact. Attach source identifiers to the observations so the responder can open the log entry or change record behind a claim. Numbered excerpts with timestamps make the comparison possible without asking the model to fill gaps from imagination.

Keep those excerpts separate from the instructions about what to do with them. A command-looking line copied from a ticket is part of the material being examined, not a new order for the assistant. Here is a reusable starting prompt:

Task: Help the responder choose the next read-only check.
Service and window: [service, start, end, timezone]
Observed impact: [measurement and source ID]
Evidence: [numbered, timestamped excerpts]
Constraints: Do not execute changes or invent missing values.

Return:
- Observed facts, each linked to an evidence ID
- Plausible explanations and evidence against each
- The next read-only check and what its result would distinguish
- Missing information that could change the recommendation
If the evidence is insufficient, say so.

Look at what the requested answer gives the person on call. A fact has a place to go back to. An explanation has to sit beside something that could count against it. The next check has a purpose: comparing the earliest errors with the rollout might rule out that rollout as the event that started them. A long, impressive query that cannot change either explanation leaves the responder with more to read and no better choice.

The last line also gives the assistant somewhere useful to stop. If the rollout time is missing, it can name that missing record rather than manufacture a timestamp. Ask for a concise account of the observations and conclusions, not a transcript of the model's internal reasoning.

The prompt shapes the proposed answer. It does not set the assistant's permissions. Writing “Do not execute changes” is useful instruction, but preventing execution requires the surrounding tools and application to enforce that limit.

From incident records to a checkable answer
Arrows connect the task and supplied evidence to answer instructions, a model draft with references and unresolved questions, and responder review.
The prompt carries the question and its source material together. The proposed answer returns references and unresolved questions for the responder to examine.
Follow the question through the draft

This proposed structure connects supplied incident records and answer instructions to a draft, then to the responder who checks it. It depicts an analysis workflow, not an executed investigation. Diagram labels: Task + supplied evidence: Question, logs, timeline and source identifiers; Answer instructions: Separate observations, hypotheses and missing facts; Model draft: Findings with references and unresolved questions; Responder review: Check claims before using the draft.

Show an answer that leaves the incident open

You can put sample questions and answers beside the prompt to demonstrate what helpful looks like. This is often called few-shot prompting. The samples show more than formatting: if every example with high processor (CPU) use ends in “add capacity,” they also suggest a shortcut past other possibilities, such as a database wait or a limit in a service being called.

Our two timestamps make a better example of testing an explanation. The 14:02 errors precede the recorded 14:05 rollout. That weakens the claim that this rollout started those errors, while leaving an earlier unrecorded change and an additional problem after 14:05 open. A proposed answer could say:

“The 14:05 rollout cannot explain the start of errors at 14:02 on this record. Check for an earlier change, then compare behavior across 14:05 to see whether the rollout added a second problem.”

The answer has not solved the incident. It has made two different questions visible: what started the errors, and whether the later rollout changed them. The responder can now examine change history and compare behavior on both sides of 14:05 instead of trying to make the earlier errors fit a later event.

Not every example should leave everything unresolved. Include cases where the observations support a conclusion as well as cases that need another check. Otherwise the examples can teach an equally unhelpful habit of always asking for more information. The test cases should distinguish these situations too.

When the next check calls for a command, first establish where it would run and what it would read. For Kubernetes, the context selects the cluster and access settings; the namespace narrows the group of resources within that cluster. Even a read-only query needs those choices. Collect logs relevant to the question rather than pulling sensitive material from the whole cluster.

Try a case the prompt has not rehearsed

Once the example reads well, set it aside and try redacted incident records and constructed cases that were not used to develop the prompt. These are held-out cases: they let you see what the assistant does beyond the examples it has already been shown. Does it keep the observed times intact? Can a responder find the source of each important claim? Would the suggested check actually separate the plausible explanations?

Repeat the important cases. One convincing answer can hide variation, and speed alone tells only part of the story. Measure the time a person spends checking and correcting the answer alongside the time the assistant takes to return it. An answer that arrives quickly but sends someone through irrelevant logs may save little work.

Keep the model, prompt version, document-retrieval setup and available tools with the results. Change any of them and the next answer may behave differently. An evaluation of yesterday's setup cannot simply stand in for a changed system.

Include a source excerpt that tries to tell the assistant what to do, then examine the application's permissions separately from the wording of its reply. OWASP's prompt-injection guidance describes this risk: outside content can redirect a model, so separating that content and limiting access matter. An ordinary answer that follows the prompt does not demonstrate that prohibited actions are impossible.

These checks answer different questions. The prompt trial asks whether the assistant helps the responder make sense of the record. The permission checks ask what the integration lets it do. The responder can use a recommendation without giving the assistant authority to carry it out.

Change the explanation at the point the record changed

Before adding a consequential claim to the incident record, open its cited source and compare the relevant passage. In our example, the difference between 14:02 and 14:05 changes where the investigation goes. Keep other explanations that still fit; removing them to make a cleaner summary would hide work that remains.

If the assistant initially blamed the rollout, correct that particular step: “The errors began before rollout; reconsider that hypothesis.” Save the observation and the correction together. Regenerating whole answers until one sounds right makes it harder for the next responder to see what changed the conclusion.

When the observations are spread across documents, the NotebookLM dossier guide develops the source-comparison work. If the assistant will execute anything, that is a separate problem for an enforced skill contract, which sets what its tools may do. A good answer cannot grant itself production access.

The amount of citation work should fit the consequence. A low-consequence draft may need less than a recommendation that changes an operational decision; both still need enforced permissions. The aim is to give the responder enough to judge this answer, not to make every draft carry the same research burden.

The useful answer in our hypothetical incident now has a direction: look earlier for the start of the errors, then find out whether 14:05 added another problem. As the responder brings back observations, revise the affected explanation. Those three awkward minutes have become something the investigation can use.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next