What prompt caching saves in an incident-handoff workflow

The handoff assistant reads the same response instructions and service reference before each new incident update. Only the latest evidence changes. Prompt caching can reuse some work on that repeated beginning. The next question is whether the cheaper request still gives the reviewer a handoff they can use.

For this assistant, the useful comparison is what it costs to produce a handoff the reviewer accepts, including retrieval, failed attempts, retries and verification. This applies the task-budget reasoning in our guide to AI token usage to a specific engineering choice. The pilot below follows a read-only incident handoff from prompt assembly through the bill and the reviewer’s decision, so a cache hit can be explained in terms of work the team actually needed.

The repeated beginning is where reuse starts

A prompt cache does not hand back yesterday’s answer. It reuses intermediate model computation for an unchanged beginning, or prefix, of the request; the model still processes new material and generates a response. In OpenAI’s prompt-caching documentation, the reusable state is described as key-value tensors and the rendered prefix must match. A relevant change before the cache boundary can prevent reuse. Keeping requests in the same session is not enough by itself.

Imagine two handoffs made a few minutes apart. The instructions and service reference have not changed; the incident evidence has. Their shared beginning is the work a prefix cache may reuse. Put a changing timestamp at the front, however, and the match can end before the useful repeated material begins. The topic of the conversation is less important than the actual order of its contents.

For the handoff assistant, a possible layout puts approved instructions, tool definitions and a versioned service reference in that stable prefix. The incident question and fresh evidence follow as the changing suffix. This is an application design to evaluate, rather than a request format shared by every provider. Rearranging material must preserve roles and instruction priority, because obtaining reuse is not a reason to change which instructions govern the task.

An old inventory is not made current by being cheap to read. Keep a deployment snapshot that changes every minute with the fresh incident evidence. A service reference may be suitable for reuse if the application first selects its current approved version. That small distinction protects both goals: reuse work that really repeats and give the assistant the facts that describe this incident.

What would count as a usable handoff?

Describe the handoff the reviewer needs: affected service, unresolved questions, supporting observations and no recovery claim beyond the evidence. Apply that same standard to both configurations. A lower bill paired with a more forgiving review would not show the cache helped.

In a hypothetical handoff pilot, both configurations receive 100 tasks. The baseline costs $12 and yields 80 accepted handoffs: $0.15 for each usable result. The caching candidate costs $8 but yields only 40: $0.20 per accepted handoff. These invented figures illustrate the arithmetic, not provider prices or measured model performance. They show how spending less overall can still make the desired output more expensive.

Those totals do not explain why the cheaper setup produced fewer accepted handoffs. The prompt, model or included evidence may have changed. Hold them steady and read the rejected outputs before giving the cache credit or blame.

A retry still costs money even when it disappears from the final transcript. Keep every attempt in the spending total, while counting the eventual accepted handoff only once. Record reviewer minutes beside that total. Turning minutes into dollars requires a stated labor-cost assumption; the token bill cannot supply one for you.

Give both configurations the same work

The dataset should represent the questions the handoff assistant actually encounters, including cases that resist a neat summary. Conflicting timestamps, missing observations, an updated service reference and an unresolved recovery check all test useful distinctions. Synthetic or appropriately sanitized evidence avoids bringing unnecessary sensitive content into the experiment. Stable task identifiers let the results be joined later, including attempts that never became an accepted handoff.

Return to the two handoffs a few minutes apart. For the comparison to mean anything, each configuration receives the same evidence for each task. The model identifier, endpoint, prompt and source-document versions, tools, generation settings and acceptance rubric stay fixed and recorded. Otherwise a better answer from the candidate might come from changed evidence rather than reuse.

The first request is part of the cost, even if it has nothing to reuse. Some APIs cache automatically, so a baseline is not necessarily uncached. A documented control can establish the intended condition where one exists; otherwise the usage records must distinguish observed cold requests from warm ones. The team cannot simply name one column “no cache” and assume the service obeyed.

Now send another eligible prefix with different incident evidence at an interval the assistant would normally encounter. Artificial warming traffic would change the economics unless its purpose and cost were included. Later, change the approved reference version or tool schema, and try beyond the documented retention period. A handoff still needs to finish with current information when reuse disappears.

At review, the task identifier reconnects that handoff to every attempt that helped or failed to produce it. Spending, rejection reasons, reviewer minutes, first-response time and total completion time describe different parts of the same job. A quick cached response that needed a second attempt is not the same outcome as a quick accepted handoff.

Timing can otherwise confuse the story. If baseline requests run during quiet hours and the candidate runs under peak demand, the latency comparison inherits that difference. Interleaving or randomizing order where practical helps. Timeouts and missing responses remain rows in the dataset, with a defined request limit and spending ceiling containing the pilot. A small sample can expose broken accounting or a missed constraint. It cannot make a rare slow request well understood just because a spreadsheet offers a percentile function. Keep the individual failed and delayed handoffs in view before using a summary statistic to support expansion.

Translate each provider’s bill before comparing it

The reviewer’s bill needs the same care as the handoff. A provider response may report several input counts that look as though they overlap. If the adapter reads them incorrectly, the saving changes before the calculator even runs. Anthropic exposes cache_creation_input_tokens, cache_read_input_tokens and input_tokens. In that accounting model, the last field is the input outside the cache read and write portions. For an invented usage row with 100 ordinary input tokens, 200 cache-creation tokens and 700 cache-read tokens, total input is 1,000 tokens. These are illustrative counts, not charges or a provider response collected here. The three fields add together; cached tokens must not be subtracted again from the ordinary input field. Anthropic’s usage definitions are therefore part of implementing the adapter, rather than an optional footnote to the savings chart.

The same request cannot assume the same cache controls when the team changes providers. The Gemini caching guide distinguishes implicit reuse from explicitly managed cache objects. The Interactions API supports implicit caching. Explicit cache objects are available through generateContent on supported Gemini 3 and 2.5 models; Gemini 4 Argon and later models do not support that explicit route. A pilot must follow the endpoint it actually uses; a feature with the same general name may not offer the same control over reuse.

Our invented 1,000-token row still cannot supply a dollar total. The adapter must retain the provider response and put its usage into non-overlapping charge categories, using rates for the actual model, service tier and region. Applicable retention, storage and tool charges belong there too. OpenAI’s documented cache-write rate, for example, replaces the ordinary input rate for those tokens rather than adding a second charge. Without that distinction, a supposedly careful ledger can double-count the very work it was built to measure.

Matching a small ledger to provider billing is how the team finds those mistakes before expanding the handoff workflow. A cache-read count records behavior, not necessarily its charge. If one call times out without usage, its charge is unknown, not zero. The price snapshot and calculation version keep a later rate change from silently rewriting this comparison.

A local calculator for incomplete cost records

The Python ledger calculator and illustrative sample ledger provide a local arithmetic check using only Python’s standard library, without network calls. Their boundary is deliberately narrow: they take normalized USD charges, not raw provider usage. They do not retrieve prices or decide whether a handoff deserves acceptance.

python3 prompt-cache-ledger.py prompt-cache-sample.json

Suppose a handoff needs two attempts before review accepts it. Its task record has one final boolean acceptance decision but both attempts, each with a unique identifier and attributable charge, including the tool costs measured by the experiment. If the first attempt’s bill is missing, its charge is null, not silently omitted. Shared cache preparation or storage belongs once in group-level overhead, not again under every handoff. Reviewer minutes remain a separate measure of the effort needed to get that acceptance.

The sample reproduces the hypothetical handoff calculation: $0.15 per accepted baseline task and $0.20 per accepted candidate task. It also includes a third group with a missing charge, whose cost-per-accepted result is null while its known subtotal remains available. Zero accepted tasks likewise produces an undefined result rather than zero cost per success. Those are useful refusals to invent an answer, not defects to erase from the report.

There is one absence the calculator cannot detect: the failed first attempt never entered into its input at all. Matching attempt identifiers against source records catches that omission, including failed or abandoned tasks. This does not require copying sensitive prompts into the ledger. Identifiers and charges are enough for the accounting; evidence bundles can remain within their existing access and retention controls, linked where review needs them.

When caching is worth maintaining

The reviewer still needs a usable handoff at the right time. A delay that suits an overnight summary may be unacceptable to the responder waiting during an active incident. Quality, waiting time and spending therefore decide whether the cache earns its upkeep; a high hit rate cannot answer for them.

When acceptance cannot be decided repeatably, cost per accepted task is not a sound basis for expanding the workflow. Keep the work exploratory and report findings and review effort until a suitable outcome measure exists. Similarly, prefixes that seldom repeat within the usable retention window may justify a simpler configuration. Manufacturing repetition would improve the experiment’s appearance while answering a different operational question.

A promising result can move into a bounded share of eligible read-only work, with the earlier prompt assembly and cache configuration available for reversal. Reverting must still use current evidence and preserve tool restrictions; it should not resurrect an old service reference. Changes to the model, policy or schema also call for another behavior check before the previous result is carried forward.

Bring the two handoffs back beside the bill. The repeated beginning may have cost less, but the fresh incident evidence still had to arrive and the reviewer still had to accept the result. The cold first request, any retry and any reference revision now have a place in the account instead of disappearing behind a cache-hit percentage. The agent evaluation guide develops that acceptance decision. Keep caching where the complete job benefits; choose simpler assembly where ordinary requests barely repeat.