On this page5 sections
An agent sends the same instructions and conversation history twice, with a different question at the end each time. Much of the beginning could be reused. Now imagine the second request starts with a fresh timestamp. To us, almost everything is unchanged. To a cache looking for an exact shared beginning, the requests disagree before they reach the familiar material.
That beginning is the prompt prefix. Prompt caching can reuse it when it matches; it does not skip ahead looking for familiar paragraphs farther down. OpenAI’s guide describes this repeated-prefix behavior, subject to eligibility and retention rules. It explains why a small timestamp can affect more processing than a large amount of unchanged history.
Compare what was actually sent: developer messages, tools, retained history and the new turn. A conversation ID does not make those contents identical. Depending on the model, cache boundaries may be automatic or explicitly marked. When reuse qualifies, it can reduce the processing charge for input tokens, the pieces into which the model divides text and other input. Whether that saves money on the job takes one more comparison.
Keep the model bill separate from the cost of getting the work accepted. A repair can use cheaper cached input and still require more retries, tool work or review. Compare the same completed tasks before deciding that the lower input charge improved the workflow.
Let the large request count as a large request
Take two illustrative eligible requests. One sends 2,560 tokens and reuses 2,048, or 80%. The other sends 23,040 and reuses 2,304, or 10%. Average those percentages and you get 45%. It looks tidy, but it gives the small request the same influence as the one carrying nine times as much input.
Add the tokens instead. The pair reused 4,352 out of 25,600 input tokens: 17%. That tells us how much input was reused across the pair. Keep the individual 80% and 10% as well, because they show where the benefit occurred. For a defined population of eligible requests, the overall ratio is total cached input tokens divided by total eligible input tokens.
Keep those request-level ratios when comparing workflows, models, prompt versions and session ages. Retain cache-write counts when the usage report provides them. They can explain why two calls behaved differently without quietly giving a tiny call the same weight as a large one in the total.
Long, expensive sessions can lose reuse while a fleet-wide average looks healthy. Separate starts, tool changes, compaction, which shortens retained history, and resumed sessions. Then the team can inspect what changed in the affected requests instead of trying to make every kind of conversation fit one target percentage.
There is another trap in a good-looking ratio: irrelevant history can cache beautifully. Removing it may reduce the hit ratio and still reduce total input and work. Read cached tokens beside total tokens and task results. Otherwise the metric gives us an incentive to keep sending material nobody needs.
Look for the first disagreement
Put the two rendered requests around a miss side by side. Yesterday’s begins with policy; today’s begins with a new timestamp. Or the same files appear in a different order. Either gives you a concrete place to start. If the beginning really matches, check the model and endpoint’s minimum length, retention period and boundary rules. Matching text must still qualify for caching.
For a long request, give its policy, tools, repository context, retained history and changing tail separate hashes: compact digests that change when their input changes. A different digest points to the segment worth inspecting without logging the full request. Treat these values as protected telemetry, using keyed or suitably salted hashing where needed, and keep secrets out of the record.
Putting the timestamp after stable instructions may preserve a longer shared beginning. Sorting an accidentally shuffled file or tool list may help too. Preserve meaning when doing either. An instruction that qualifies a later task must still apply to that task after rearrangement. Replay the work and check both the cached-token count and the answer, rather than accepting a better cache number on its own.
A cache key helps label related traffic. It cannot make different early content match. Keep the actual returned cached-token counts beside the request construction that produced them; the key alone does not establish reuse.
Notice what the host changed between turns
A restart, shortened history, a different repository or a newly added tool can reshape the next request. Record that event alongside the cache counts. A drop in reuse then comes with a possible explanation instead of becoming another unexplained line on a chart.
Suppose a hypothetical repair agent loses reuse after turn six. Its host places a fresh file list before the stable policy on every turn, and the list’s order varies. In an illustrative controlled replay, sorting that list and moving the changing worktree details later restores a longer reusable beginning. This is an example of the mechanism, not a measured OpenAI result.
A real repository change can make a cache miss the right outcome. The request needs current files, even if yesterday’s version would reuse more input. Check whether fewer mistakes and corrections justify the new processing. A cached assumption about code that no longer exists is not a saving worth protecting.
The supervised worker guide puts restart, cancellation and uncertain outcomes with the host. Request construction and cache counts belong there too. Then a restart that rebuilds the input differently is visible in the same history as the task it affected.
Follow the saving through the correction
Calculate model charges from actual usage fields and current prices. Track elapsed time, retries, tools, reviewer time and failed attempts in a separate task budget. The token guide follows those costs past the final successful response.
Imagine a repair whose first answer is cheap because much of its input was cached. It misses a condition, so another call is needed and a reviewer spends time correcting the patch. Keep the retry’s cached and uncached input, reported cache writes, output and tool time with the original task. Now the bill and review effort can show whether the initial saving survived the correction.
A repetitive review and a research agent continually reading new documents will not necessarily have the same useful hit rate. Establish the workflow’s normal pattern, then investigate unexplained changes. Otherwise we risk making the work more repetitive just to make the cache look busier.
Keep the useful change, not yesterday’s input
For our shuffled file list, the comparison should show which segment changed, how total and cached input moved, and what happened to completion and review effort. Sorting accidental churn may help all three. Keeping an obsolete repository snapshot is a different decision, even if its cache ratio is excellent.
New policy, tools or source material sometimes belong at the cost of a deliberate miss. Record that reason beside the counts. The next person can then distinguish necessary fresh input from a timestamp or shuffled list that spoiled reuse for no useful reason. That is where the cache work can help without changing the job it was meant to serve.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- prompt-caching guidedevelopers.openai.com
