What prompt caching saves in an incident-handoff workflow

A handoff assistant has a repetitive job before it ever reaches the new incident. It reads the same response instructions, the same tool descriptions and perhaps the same service reference, then finally gets to the evidence that changed. Prompt caching can spare some of that repeated computation. Whether it saves the team money depends on what happens to the handoff after the cheaper request returns.

For this assistant, the useful comparison is what it costs to produce a handoff the reviewer accepts, including retrieval, failed attempts, retries and verification. This applies the task-budget reasoning in our guide to AI token usage to a specific engineering choice. The pilot below follows a read-only incident handoff from prompt assembly through the bill and the reviewer’s decision, so a cache hit can be explained in terms of work the team actually needed.

The repeated beginning is where reuse starts

A prompt cache does not hand back yesterday’s answer. It reuses intermediate model computation for an unchanged beginning, or prefix, of the request; the model still processes new material and generates a response. In OpenAI’s prompt-caching documentation, the reusable state is described as key-value tensors and the rendered prefix must match. A relevant change before the cache boundary can prevent reuse. Keeping requests in the same session is not enough by itself.

For the handoff assistant, a possible layout puts approved instructions, tool definitions and a versioned service reference in that stable prefix. The incident question and fresh evidence follow as the changing suffix. This is an application design to evaluate, rather than a request format shared by every provider. Rearranging material must preserve roles and instruction priority, because obtaining reuse is not a reason to change which instructions govern the task.

The word “stable” deserves some care. A deployment inventory refreshed every minute belongs with current evidence even if repeating the old inventory would improve the hit rate. A service reference can be a candidate for reuse if the application selects its current approved version before each request. That makes freshness part of assembly, rather than something a reviewer has to discover after reading an outdated handoff.

What would count as a usable handoff?

Before comparing configurations, decide what the reviewer needs from this one workflow. A handoff should identify the affected service, preserve unresolved uncertainty, point to its supporting observations and avoid declaring recovery without evidence. Applying the same acceptance criteria to every group gives the experiment a consistent result to count. Otherwise, a lower bill and a more forgiving reviewer can look like the same improvement.

In a hypothetical handoff pilot, both configurations receive 100 tasks. The baseline costs $12 and yields 80 accepted handoffs: $0.15 for each usable result. The caching candidate costs $8 but yields only 40: $0.20 per accepted handoff. These invented figures illustrate the arithmetic, not provider prices or measured model performance. They show how spending less overall can still make the desired output more expensive.

The next question is why acceptance changed. The figures do not establish that caching damaged quality. A changed prompt, a different model or truncated evidence could explain the difference. Keeping those inputs fixed, then inspecting rejected work, lets the team investigate the cache-related choice rather than assign causality to whichever setting has the most convenient name.

Every attempt stays in the numerator of this calculation. A failed request consumed resources, and three revisions of the same handoff still end in at most one accepted task. Reviewer minutes remain alongside API spending as a separate measure. Converting that time into money would need an explicit labor-cost assumption; token accounting alone cannot provide it.

Give both configurations the same work

The dataset should represent the questions the handoff assistant actually encounters, including cases that resist a neat summary. Conflicting timestamps, missing observations, an updated service reference and an unresolved recovery check all test useful distinctions. Synthetic or appropriately sanitized evidence avoids bringing unnecessary sensitive content into the experiment. Stable task identifiers let the results be joined later, including attempts that never became an accepted handoff.

There are five parts to the comparison. Their purpose is to make a difference interpretable, not to produce the highest possible cache-hit number.

1. Fix the surrounding configuration. Record the model identifier, endpoint, prompt and source-document versions, tool definitions, generation settings and acceptance rubric. Holding these steady makes a change in cache behavior easier to separate from a change in the task itself.
2. Observe the current path. Some APIs cache automatically, so “baseline” does not necessarily mean uncached. Use a documented control where one exists. Otherwise retain reported usage and describe observed cold and warm cohorts honestly.
3. Exercise ordinary repetition. Repeat eligible prefixes with different task evidence at realistic intervals. The first request belongs in the results too. Artificial warming traffic only belongs in the design if its purpose and cost are counted.
4. Let the inputs change. Update the reference version and tool schema, and test beyond the documented retention period. The application still needs to send current information and complete its task when reuse disappears.
5. Close the loop at acceptance. Join all attempts to the final review result. Spending, rejection reasons, reviewer minutes, time to first response and total completion time then describe different parts of the same work.

Timing can otherwise confuse the story. If baseline requests run during quiet hours and the candidate runs under peak demand, the latency comparison inherits that difference. Interleaving or randomizing order where practical helps. Timeouts and missing responses remain rows in the dataset, with a defined request limit and spending ceiling containing the pilot. A small sample can uncover broken accounting; it cannot support a dependable production percentile simply because the spreadsheet calculates one.

Translate each provider’s bill before comparing it

Usage fields can resemble each other while counting different things. Anthropic exposes cache_creation_input_tokens, cache_read_input_tokens and input_tokens. In that accounting model, the last field is the input outside the cache read and write portions. Total input is the sum of all three, not the last field with cached tokens subtracted again. Anthropic’s usage definitions are therefore part of implementing the adapter, rather than an optional footnote to the savings chart.

Controls vary as well. The Gemini caching guide distinguishes implicit reuse from explicitly managed cache objects. The Interactions API supports implicit caching, while explicit caching requires generateContent. A pilot must follow the endpoint it actually uses; a feature with the same general name may not offer the same control over reuse.

The adapter should preserve the provider response and map it into non-overlapping charge categories. Rates need to match the actual model, service tier and region, with applicable retention, storage and tool charges included. OpenAI’s documented cache-write rate, for example, replaces the ordinary input rate for those tokens rather than adding a second charge. Without that distinction, a supposedly careful ledger can double-count the very work it was built to measure.

A small reconciliation against provider billing is a useful test of this translation before totals become a rollout argument. Cache-read counts explain behavior; they do not by themselves establish charges. A timed-out attempt with no usage record remains an unknown charge. Keeping the price snapshot and calculation version with the run also stops a later rate change from silently rewriting what this experiment cost.

A local calculator for incomplete cost records

The Python ledger calculator and illustrative sample ledger provide a local arithmetic check using only Python’s standard library, without network calls. Their boundary is deliberately narrow: they take normalized USD charges, not raw provider usage. They do not retrieve prices or decide whether a handoff deserves acceptance.

python3 prompt-cache-ledger.py prompt-cache-sample.json

Each task record has a final boolean acceptance decision and its list of attempts. Every attempt needs a unique identifier and the attributable charge, including whichever tool costs the experiment measures. An unknown charge is represented by null. Shared cache preparation or storage belongs once in group-level overhead, since repeating it for every task would inflate the comparison. Reviewer minutes are retained separately.

The sample reproduces the hypothetical handoff calculation: $0.15 per accepted baseline task and $0.20 per accepted candidate task. It also includes a third group with a missing charge, whose cost-per-accepted result is null while its known subtotal remains available. Zero accepted tasks likewise produces an undefined result rather than zero cost per success. Those are useful refusals to invent an answer, not defects to erase from the report.

The calculator cannot notice attempts that never reached its input. Reconciliation therefore starts upstream, matching attempt identifiers against source records and retaining failed or abandoned tasks. The ledger needs those identifiers and charges, rather than copied sensitive prompts. Evidence bundles can stay within existing access and retention controls, linked where the review requires them.

When caching is worth maintaining

A cache configuration adds something to operate, so the pilot needs a reason to keep it. Set a quality floor, an acceptable completion delay, a spending ceiling and a minimum worthwhile improvement from the workflow’s requirements. An incident handoff and an overnight summary have different consequences for delay. Neither gains much from borrowing a universal hit-rate target.

When acceptance cannot be decided repeatably, cost per accepted task is not a sound basis for expanding the workflow. Keep the work exploratory and report findings and review effort until a suitable outcome measure exists. Similarly, prefixes that seldom repeat within the usable retention window may justify a simpler configuration. Manufacturing repetition would improve the experiment’s appearance while answering a different operational question.

A promising result can move into a bounded share of eligible read-only work, with the earlier prompt assembly and cache configuration available for reversal. Reverting must still use current evidence and preserve tool restrictions; it should not resurrect an old service reference. Changes to the model, policy or schema also call for another behavior check before the previous result is carried forward.

For the handoff assistant, the decision record should connect reused prefixes and fresh incident evidence to the handoffs reviewers accepted, counting cold requests, revisions, unresolved charges and reviewer effort. The incident-response agent evaluation guide develops the acceptance question in more detail. If that complete comparison shows a worthwhile saving, the team has a reason to maintain caching; if ordinary work supplies little repeated material, simpler prompt assembly may be preferable.