Choosing an AI model: count the work it leaves you

In this invented comparison, configuration B wins the race to put an answer on the screen: a median of two seconds, against four for A. Then someone has to read the answer. Checking takes a median of forty-five seconds for B and fifteen for A, and one of B's answers contains an unsupported instruction.

The faster reply has not finished the job. It has passed more of it to the reader.

These are fictional results for the Orchard document assistant, not measurements of any product. They illustrate what a model comparison needs to include: the work required to reach an answer someone can use, and the restrictions on where the documents may go. The model name alone tells only part of that story. Instructions define the task, retrieval brings in the source material, and output limits can cut an answer short. Record the whole setup you are comparing.

What would make this answer worth using?

Orchard's assistant answers four questions from a supplied service note. It needs to distinguish an accepted request from a completed export, preserve what happens to queued work during a restart, and say when the note leaves a fact unknown. A confident answer that changes one of those relationships can send its reader in the wrong direction.

For this exercise, imagine the team wants a median response time within ten seconds and a median checking time no greater than thirty seconds across the test set. To find the median, order the measurements by size and take the middle value, or the average of the two middle values when there is an even number. These targets belong to the example; they are not recommended deadlines for every service.

There is another question to settle before either stopwatch matters. Are these configurations allowed to receive the documents? If the documents must remain in an approved environment, a destination outside it is not a candidate. A local model may fit that restriction, but its application's logs, access and resource use still need attention.

The team also needs to know which mistakes it cannot accept. “The note does not say” can be an honest answer to a missing retention period. An invented instruction to delete queued work is another matter. A collection of easy correct answers should not conceal that failure in an average.

Keep the clock running after the reply

The evaluation sheet from the previous lesson gives this comparison a starting point. Bring it, two configurations you are allowed to use, and a way to record elapsed time. Give both setups the same questions and source text. If timing by hand, alternate their order, and keep the failures and retries as well as the successes. Otherwise the comparison can quietly become a display of each system's best attempt.

Invented comparison result	Configuration A	Configuration B
Answers supported by the source	4 of 4	3 of 4
Unsupported operational claims	0	1
Median time to a model response	4 seconds	2 seconds
Median time spent checking	15 seconds	45 seconds
Cost per request	Record actual usage and current rates.	Record actual usage and current rates.

A fits both targets in this invented result. B fits the response-time target but misses the checking-time target and includes an unsupported operational claim. The person using B gets a reply sooner, then spends longer finding out whether it can be used. The four-second reply is looking less slow.

It is tempting to add each pair of medians and declare a total time. That would give a number the measurements do not establish. The answer in the middle of the response-time list need not be the one in the middle of the checking-time list. Adding those two medians therefore does not give the median time to completed work. If that is the statistic you need, record total elapsed time for each task. The separate timings still help locate where the reader's effort goes.

Cost needs the same wider view. A low-priced request may need retries and a longer review before its answer is usable. For a hosted model, keep actual usage with the provider's current rates. For a local runtime, record hardware and demand level alongside answer quality. One request on an idle machine cannot show how the setup will behave when several people need it at once.

A small hypothetical calculation makes the difference visible. Suppose twenty requests cost $0.04 altogether, but only sixteen answers pass the agreed checks. The request charge per checked answer is $0.04 divided by 16, or $0.0025. Dividing by twenty would count the four unusable answers as if they had completed the job.

That figure uses invented prices and includes request charges only. Keep checking time beside it. It does not price every part of using the system, and it cannot make a disqualifying error acceptable merely because the number is small.

Choose the next setup, and know what would change your mind

For these fictional results, the decision can be specific: A is the initial setup because all supplied answers were supported and median checking time met the trial target. B needs its unsupported instruction repaired before reconsideration. Another engineer can see both why A won and what B would need to improve.

The answer sheet can also reveal that neither model is the first thing to replace. Suppose both miss the retention question because the source passage never reached them. A larger model would still be missing that passage. Repair the route by which the information reaches the model and repeat the test. If both received the correct passage and reversed it, the next investigation concerns the instructions and how the models handle the task.

Keep the criteria, exact inputs, raw answers, checking effort, data restrictions and rejected alternative together. Include what would reopen the choice, perhaps longer documents, different privacy requirements or a changed rate. The comparison then remains useful when the work changes, rather than becoming a recommendation whose reasons everyone has forgotten.

This small exercise chooses a next experiment, not a production guarantee. Representative tasks, load and failure handling still need examination. What it has already shown is where the reader's time goes. When B's reply appears first, there is still a person checking it. A useful comparison follows that person far enough to find out which setup actually leaves less work.

Sources and further reading

OpenAI's model evaluation guidance supports comparisons designed around the particular task. Before a real purchase, consult the selected provider's current documentation for prices, limits and data handling. No provider's performance or pricing is measured by this fictional exercise.