Evaluating AI answers: when a convincing fix is the wrong fix

In this fictional example, an assistant offers a tidy fix: “Restart the API and the export queue will clear.” The sentence sounds useful because it names an action and promises a result. Unfortunately, the service does not work that way.

The useful question is what made the answer wrong, and how to catch that same mistake after the assistant changes. A small set of questions, checked against a source you understand, can tell you considerably more than whether the response sounded convincing. For this exercise, a text editor is enough. A model interface you already use is optional; no credentials or subscription are needed for a first pass on paper.

The file is still waiting to be made

Our invented service, Orchard, accepts a request to export a file and puts the job on a waiting list, or queue. A worker picks it up later and makes the file. The API, the interface receiving the request, can reply before that work is finished. “Accepted” is good news, but it is not yet a file someone can download.

Here is the complete source note for the exercise: “The API accepts work. Workers create exports. Restarting an API process does not empty the queue. Only a worker can finish an export. The note does not specify retention.”

Before asking the assistant anything, the note already gives us a few firm answers. The worker finishes the export. Restarting the API does not empty the queue. Retention, how long the exports are kept, is left unspecified. Those distinctions give the assistant's prose something to answer to.

Question	What the note supports	What would fail
Does API success prove completion?	No. Accepting the request is different from finishing the export.	Claiming that the file is already ready.
Will restarting the API clear queued work?	No. The queue remains, according to the note.	Claiming that a restart removes queued exports.
How long are exports retained?	The note does not give a retention period.	Supplying an invented period.
Which component completes an export?	The worker makes the file.	Giving that job to the API or database without support.

The wording can vary. “The background worker makes the file” and “A worker completes the export” say the same useful thing here. Requiring an exact copy of the answer key would measure the assistant's ability to reproduce wording, rather than preserve the relationship you care about.

Wrong, unknown and beside the point

Give the assistant all four questions with this instruction: “Answer only from the supplied note. Identify the sentence supporting each answer. Say when the note does not establish the answer.” Save the exact input and response, plus the model and settings when available. That keeps a later comparison from depending on a memory of what you think you asked.

Our opening recommendation, “Restarting the API empties the queue,” directly conflicts with the note. Mark it contradicted. “Exports remain for seven days” has a different problem: the note gives no period, so that claim is unsupported. An answer is supported when the note establishes it. A contradiction calls for correcting a false relationship; an unsupported answer may need information the source never supplied.

There is a third wrinkle. Ask how long files remain, and “Workers make files” is true but no help at all. Record whether the answer actually addressed the question as well as whether its claim was supported. Otherwise an assistant could do remarkably well by offering a collection of unrelated truths.

Follow its citation to the specific sentence, too. A link to a real document does not tell you that the document said what the answer claims. The sentence you reach should support the particular relationship being asserted.

Did the note reach the model?

Before changing the assistant, look at the actual material sent to it. If Orchard's note never arrived, the problem may be in retrieval, the step that finds source material, or in assembling the request. If the note arrived intact and the answer still reversed it, investigate how the system used the information. The same wrong recommendation can have quite different beginnings.

One row per question is enough to keep the investigation legible: source passage, answer, evidence label, whether it answered the question and the correction needed. Include the time spent checking and repairing it. A reader who spends two minutes untangling an answer has been handed a different amount of work from one who can verify it in ten seconds.

Repeat the questions before changing the setup. If the answers vary, that variation belongs in the result. Then change one ingredient, perhaps the instruction or the retrieved passage, and compare again. Replacing the model, prompt and documents all at once may improve the answer, but it leaves you guessing which change helped.

“Is it ready?” needs one more detail

Now imagine someone asking Orchard, “Is it ready?” Even a perfect copy of the service note cannot tell the assistant which export the person means. Asking for that identity can be the useful response. Keep the question in your test so you can see whether the assistant notices what is missing.

“How much work is in the queue right now?” exposes another limit. The note describes how Orchard works. It does not observe the current queue. An assistant that invents a live count from this document has crossed from explaining the service into reporting something it has not been shown.

A small test with recognizable mistakes

The results are more useful when their differences remain visible. Report supported, contradicted and unsupported answers separately, alongside unanswered questions, missing context and checking time. Keep the actual examples beside the totals. Another reader can then examine a disagreement rather than argue with an unexplained score.

Four questions cannot establish accuracy for every document or user. They can catch our false restart advice, invented retention period and ambiguous request again after the setup changes. The next time an answer promises to clear Orchard's queue, you have a way to find out whether the assistant understood the service, and how much checking it still leaves to the person who needs the export.

Sources and further reading

OpenAI's Evaluation best practices discusses tests built around a particular task and improved through repeated comparisons. Orchard's service note, answers and scoring exercise are synthetic teaching material, not a product benchmark.