Measuring prompt-cache reuse in long agent sessions

In this illustrative example, an agent sends the same instructions and conversation history twice, followed by a different question each time. Much of that beginning may be reusable. Now put a fresh timestamp at the very top of the second request. Almost everything still looks familiar to you, but the two requests disagree before the useful shared material even starts.

Prompt caching can reuse an exact beginning of a request, called its prefix. It does not search farther down for familiar paragraphs. That is why one small timestamp can matter more than pages of unchanged history. OpenAI's guide describes this repeated-prefix reuse; eligibility and retention rules still apply.

The comparison uses the request actually sent to the model, including developer messages, tools, history and the new turn. Keeping a conversation ID does not make those contents identical. Depending on the model, cache boundaries can be automatic or explicitly marked. When reuse does qualify, the agent may avoid paying the same processing cost for every input token. The question is whether that saving survives the rest of the job.

Keep model charges separate from the work of operating the repair workflow. Cached input may cost less while retries, tools and review consume the saving. Compare the same completed tasks before calling the lower token bill an improvement.

Measure the cacheable prefix

Take an illustrative pair of eligible requests. The small one sends 2,560 tokens and reuses 2,048: 80%. The much larger one sends 23,040 and reuses 2,304: 10%. Put the two percentages side by side and 45% looks like a reasonable average. But that gives the small request as much say as the one carrying nine times the input.

Count the tokens instead. Together, the requests reused 4,352 of 25,600 input tokens, or 17%. That answers how much of their input was reused. The 80% and 10% still matter: they show which request benefited. For a defined group of eligible requests, the calculation is total cached input tokens divided by total eligible input tokens.

Keep the individual ratios when comparing workflows, models, prompt versions and session ages, and preserve cache-write counts where usage reports them. Those details help explain why two calls behaved differently; they should not quietly change the weight of each call in the total.

The longest expensive sessions can miss while the fleet average looks healthy. Separate starts, tool changes, compaction and resumed sessions. The useful question is what happened to those requests, not how to make every conversation serve one percentage.

A large irrelevant history may cache well. Removing it can lower the hit ratio while reducing total input and work. Read cached tokens, total tokens and task results together so the metric does not reward carrying unnecessary baggage.

Find where the prefix started changing

Start with the two rendered requests around a miss. If yesterday’s request begins with policy but today’s starts with a new timestamp, the first disagreement is already visible. A file list whose order changes can have the same effect. If the beginning really is unchanged, inspect the model and endpoint’s minimum length, retention and boundary rules: identical text still has to qualify for reuse.

For a long request, comparing all the text by eye soon becomes tiresome. Give the policy, tools, repository context, retained history and changing tail separate digests, or hashes. A changed digest identifies the segment to inspect without logging the whole request. Protect those digests appropriately, using keyed or suitably salted hashing where needed, and keep secrets out of telemetry.

Moving the timestamp after stable instructions can preserve a longer beginning. Sorting an accidentally shuffled tool or file list can do the same. But do not move material whose position changes its meaning: an instruction that qualifies a later task must still qualify it. Replay the task after rearranging the request and check both the cached-token count and the answer.

A cache key labels related traffic; it does not replace matching content. The same key cannot make different early text identical. Keep returned cached-token counts beside the construction that produced them.

Connect cache misses to agent lifecycle events

Restarts, shortened history, changed repositories and new tools all alter an agent's next request. Record that lifecycle event with cache telemetry so a drop in reuse has an explanation to inspect.

Suppose a hypothetical repair agent's reuse falls after turn six. Its host puts a freshly generated, variably ordered file list before stable policy on every turn. In a controlled replay, sorting it and moving the changing worktree delta later restores reusable prefixes. This is illustrative, not a measured OpenAI result.

A changed repository can require the opposite choice: discard reuse so the request receives current evidence. Then compare whether fewer errors and corrections justify new input. Keeping stale assumptions for a hit is a poor trade.

The supervised worker guide places restart, cancellation and uncertain outcomes with the host. Add request construction and cache telemetry there, so a restart that reshapes input is visible.

Budget the complete task

Use actual usage fields and current prices for model charges. Keep wall time, retries, tools, reviewer time and failed attempts in a separate task budget. The token guide follows the work beyond the final successful response.

Follow one repair all the way to acceptance. Its first answer may be cheap because much of the input was cached. If it misses a condition, the retry adds another call and a reviewer may spend time correcting the patch. Keep that retry’s uncached and cached input, reported cache writes, output and tool time with the original task. The final bill and review effort can then show whether the saving survived the correction.

A repetitive review and a research agent reading new documents need not share a target hit rate. Establish the normal pattern for the workflow, then investigate unexplained changes. Otherwise the work starts serving the metric.

When the cache change has earned its place

For the changing file list, show which segment changed, how total and cached input moved, and what happened to completion and review effort. Sorting accidental churn may help. Preserving yesterday's repository is a different choice.

A deliberate miss can be correct when new policy, tools or evidence must enter the request. Keeping that reason with the counts separates necessary refresh from accidental churn and directs effort toward the latter.