Prompting an assistant to investigate an incident

“What caused this incident?” asks for the answer everyone wants. With an incomplete record, “What should we check next?” may help more. It asks the assistant to move the investigation forward without choosing a final cause before the evidence can distinguish one.

Start by deciding what a useful answer must let the responder do. The template below asks for a read-only investigation step and the evidence, alternatives and missing information needed to judge it.

The incident record inside the prompt

Supply the service, time window and zone, measured impact, source identifiers and constraints. Keep the source material separate from your instructions. A command-looking sentence inside a log or ticket is evidence to analyze, not a new instruction to obey.

Task: Help the responder choose the next read-only check.
Service and window: [service, start, end, timezone]
Observed impact: [measurement and source ID]
Evidence: [numbered, timestamped excerpts]
Constraints: Do not execute changes or invent missing values.

Return:
- Observed facts, each linked to an evidence ID
- Plausible explanations and evidence against each
- The next read-only check and what its result would distinguish
- Missing information that could change the recommendation
If the evidence is insufficient, say so.

The fields each have a job. A source ID lets the responder open the observation. An alternative prevents the first plausible explanation becoming the only one. A next check must distinguish something, such as whether errors began before the rollout. An impressive query that distinguishes nothing merely adds review work.

This template guides an answer; it does not enforce permissions. Saying “do not execute” cannot replace tools configured to prevent execution. Ask for a concise account of evidence and conclusions, rather than a transcript of internal reasoning.

Make the source material part of the prompt

Read diagram description

A proposed incident-analysis prompt structure. Supply the task and evidence, ask for supported findings and unknowns, then review the answer against the source material. Diagram labels: Task + supplied evidence: Question, logs, timeline and source identifiers; Answer instructions: Separate observations, hypotheses and missing facts; Model draft: Findings with references and unresolved questions; Responder review: Check claims before using the draft.

Examples can demonstrate a useful incomplete answer

Few-shot examples are sample questions and answers placed in the prompt. They teach behavior as well as format. If every high-CPU example ends in “add capacity,” they teach that leap without examining a database wait or dependency limit. Show an example that tests its explanation.

For a hypothetical deployment investigation, compare the first error time with rollout. Errors already present would weaken a claim that this rollout initiated them. A useful answer can name that missing comparison without manufacturing a decisive cause.

Make the example concrete: errors start at 14:02 UTC and the recorded rollout starts at 14:05. That weakens the claim that this rollout started the errors. It leaves room for an earlier unrecorded change or another regression after 14:05. Check change history and behavior on both sides of the rollout.

An answer could say: “The 14:05 rollout cannot explain the start of errors at 14:02 on this record. Check for an earlier change, then compare behavior across 14:05 to see whether the rollout added a second problem.” The incident remains unsolved, but its timeline no longer has to bend around the first suspect.

Do not make every example end in a question. Show both when the evidence supports a conclusion and when another observation is needed. The evaluation should distinguish those cases too.

Before producing a command, establish its environment and scope. Even a read-only Kubernetes query needs the intended context and namespace. Limit log collection to work relevant to the question rather than sweeping sensitive material from the whole cluster.

Testing proposed checks against held-out cases

Evaluate with redacted incidents and constructed cases not used to develop the prompt. Check support for claims, preservation of constraints, missing information and whether the proposed step can separate plausible explanations. Repeat important cases to expose variation hidden by one convincing answer.

Measure checking and correction time alongside response latency. Record model, prompt version, retrieval configuration and available tools. Changing any of these can change the answer, so an older evaluation may no longer describe the running system.

OWASP's prompt-injection guidance explains why instructions alone are not a security boundary. Include source text that tries to redirect the model, and examine permissions separately. Ordinary compliant-looking answers do not prove that the integration limits actions correctly.

Keep those responsibilities distinct: the prompt asks for usable evidence; the surrounding system enforces action limits. A well-explained recommendation remains useful without making obedience into authorization.

How an answer should change with new evidence

Before putting an answer into the incident record, check claims that would alter action. Open the cited source and compare the relevant passage. Keep alternatives that still fit instead of making the summary sound more certain by removing them.

Revise where new evidence changes the reasoning. “The errors began before rollout; reconsider that hypothesis” gives a specific reason to update. Save that observation and correction. Regenerating whole answers until one sounds right obscures why the conclusion moved.

For evidence across documents, the NotebookLM dossier guide develops source comparison. For execution, use a separately enforced skill contract. The quality of an answer does not let it assign itself production access.

A low-consequence draft may need less citation work than an operational decision. Scale evidence requirements to the consequence while retaining enforced permissions for both.

For our example, a useful result sends the responder toward the earlier errors and asks whether 14:05 added another problem. As observations arrive, revise the explanation at the affected step. The record then shows what the investigation learned, rather than which confident answer was generated last.