Prompt caching: measure the cost of useful work

A prompt cache can make repeated requests cheaper while the operational task remains expensive. The assistant still retrieves evidence, makes tool calls, retries incomplete answers, and asks someone to correct the result. A high cache-hit rate tells you which computation was reused. It does not tell you whether the work was worth doing.

Evaluate prompt caching against the complete cost of accepted work, including unsuccessful attempts. That applies the same task-budget principle used in our guide to AI token usage to a narrower engineering decision: whether a particular reusable prefix deserves a production rollout. The experiment below separates cache behavior from task outcomes and leaves unknown spending visible.

What the cache actually reuses

Prompt caching reuses intermediate model computation for an unchanged prefix, the beginning of a request. It is different from returning a previously saved answer. The model still processes new material and generates a response. OpenAI describes the reusable state as key-value tensors and requires the rendered prefix to match; relevant changes before a cache boundary can prevent reuse. A session alone does not guarantee a hit. See the OpenAI prompt-caching documentation.

For an operational assistant, a candidate prefix might contain approved response instructions, tool definitions, and a versioned service reference. The changing suffix contains the incident question and fresh evidence. This is a proposed application layout, not a universal provider request format. Keep roles and instruction priority intact when arranging it.

Decide what is actually stable before changing request order. A deployment inventory refreshed every minute should not become a fixed reference merely because that improves reuse. A service document can be suitable if the application selects its current approved version before every request. An old document consistently reused is still an old document.

Choose a task with a checkable finish

Start with one read-only workflow, such as drafting a service incident handoff from a supplied evidence bundle. Define acceptance before collecting results: the handoff must identify the affected service, preserve unresolved uncertainty, reference the supporting observations, and avoid claiming recovery without evidence. Reviewers should use the same criteria in every experiment group.

In a hypothetical handoff pilot, two configurations process 100 tasks each. The baseline spends $12 and produces 80 accepted handoffs. The cache candidate spends $8 and produces 40. Its model spending is lower, but cost per accepted handoff rises from $0.15 to $0.20. These are invented arithmetic examples, not provider prices or measured model results.

This would not prove that caching harmed quality. Perhaps the candidate also changed the prompt, truncated evidence, or used a different model. It does show why a cheaper bill alone cannot approve the change. Hold those factors steady, inspect rejected work, and identify which part of the experiment changed.

Count the cost of every attempt before dividing by the number of accepted tasks. Do not remove failed requests from the spending total or count three revisions of one handoff as three successful tasks. Track reviewer minutes separately. Combining human time and API charges into money requires an explicit labor-cost assumption, which a token ledger cannot supply.

Run the comparison without hiding the cold path

Prepare a fixed, permitted dataset containing both routine handoffs and difficult cases: conflicting timestamps, missing evidence, a changed service reference, and an unresolved recovery check. Use synthetic or appropriately sanitized material. Retain task identifiers so each configuration sees the same cases. A small pilot can reveal accounting defects; it cannot establish a dependable production percentile from a handful of requests.

1. Freeze the experiment. Record the model identifier, provider endpoint, prompt version, tool definitions, source-document version, generation settings, and acceptance rubric. Change one cache-related choice at a time.
2. Measure the current configuration. Capture actual reported cache usage rather than assuming the baseline is uncached. Some APIs cache automatically. Use a documented control when available; otherwise describe the baseline honestly and compare observed cold and warm cohorts.
3. Exercise reuse. Send eligible repeated prefixes with different task evidence at realistic intervals. Measure the first request as well as subsequent requests. Avoid artificial warming traffic unless its cost and operational purpose are included.
4. Exercise change. Revise the reference version, alter the tool schema, and test after the documented retention period. Confirm that the application sends current information and remains usable when reuse falls away.
5. Review the complete task. Join all attempts to the final acceptance result. Compare spending, rejected-task reasons, reviewer minutes, time to first response, and end-to-end completion time.

Interleave configurations or randomize their order where practical. Running all baseline cases during quiet hours and every candidate case during peak demand makes latency harder to interpret. Record timeouts and missing responses instead of deleting those rows. Keep a defined request limit and spending ceiling so an accounting experiment cannot grow without notice.

Normalize usage before calculating savings

Providers expose different controls and accounting fields. Anthropic separates cache_creation_input_tokens, cache_read_input_tokens, and input_tokens; its last field represents input outside the cache read/write portions. Adding those three produces total input for that accounting model. Do not interpret the last field as total input and then subtract cached tokens again. Consult Anthropic’s usage definitions.

Google’s Gemini caching guide distinguishes implicit caching from explicitly managed cache objects. Its Interactions API supports implicit caching, while explicit caching requires the generateContent API. Choose an integration based on the endpoint you actually use; a similarly named feature does not establish equivalent controls.

For each provider adapter, preserve the original usage response and map it into non-overlapping charge categories. Check current rates, model, service tier, region, retention terms, and any tool or storage charges. OpenAI’s documented cache-write rate replaces the ordinary input rate for those tokens rather than being added on top. Do not carry that convention into another billing model without checking.

Reconcile calculated charges with provider billing for a small sample before trusting aggregate savings. A timeout without usage is an unknown charge, not a free request. Likewise, a cache-read count is a diagnostic observation, not a bill. Keep the rate snapshot and calculation version beside the run so a later price change does not silently rewrite the experiment.

Use a ledger that keeps failed work visible

Download the Python ledger calculator and illustrative sample ledger. They run locally using the Python standard library and make no network calls. The calculator accepts already-normalized USD charges; it does not fetch usage, estimate token prices, or prove that a model performed the task correctly.

python3 prompt-cache-ledger.py prompt-cache-sample.json

The input has one record per task, a final boolean acceptance decision, and a list of attempts. Each attempt includes a unique identifier and its total attributable charge, including the tool costs you have chosen to measure. Use null when that charge is unknown. Record shared cache preparation or storage costs once in the group-level overhead field, not once per task. The file also preserves reviewer minutes as a separate total.

The sample returns $0.15 per accepted baseline task and $0.20 per accepted candidate task. A third group contains a missing charge, so its cost-per-accepted result is null. The known subtotal remains available for investigation. With zero accepted tasks, the result is also undefined rather than zero. Those outcomes should keep an incomplete measurement out of a savings claim.

The deliberately small schema does not enforce completeness against a provider account. Your collection process must reconcile attempt identifiers, include failed and abandoned tasks, and prevent missing attempts before creating the file. Keep sensitive prompts out of this ledger. Store any evidence bundles under your existing access and retention controls.

Make the rollout decision explicit

Before the pilot, name the quality floor, acceptable completion delay, spending ceiling, and minimum improvement that justify the extra complexity. Choose values from the workflow’s actual requirements. A latency-sensitive incident assistant and an overnight document summarizer need different limits; neither should borrow a universal cache-hit target.

If the workflow lacks a repeatable acceptance decision, do not use cost per accepted task to approve expansion. Keep the experiment exploratory, report useful findings and review effort, and establish a suitable outcome measure first. If request prefixes rarely repeat within the usable retention window, simplify the configuration rather than manufacturing repetitive traffic to make the chart improve.

For a promising result, roll out to a bounded portion of eligible read-only work. Keep the previous prompt assembly and cache configuration available for reversal. Reverting must preserve current evidence and tool restrictions; it must not revive a superseded service reference. Verify behavior after a model, policy, or schema change before carrying forward the earlier result.

The practical deliverable is a short decision record: the exact configuration tested, observed reuse, complete spending, accepted and rejected outcomes, unknown charges, reviewer effort, and cold-path behavior. Our incident-response agent evaluation guide develops the separate question of whether the result deserves acceptance. Use both records before expanding the workload.