Tokens and context windows: what stays in the conversation

Scroll up through a long conversation with an AI assistant. There is the plan, the correction to the plan, and the small exception you added after lunch. You can still see all three. It is natural to assume the model can, too.

What matters, though, is what the application sends on the next request. The model receives a limited sequence of small units called tokens. That limit is its context window. A long chat can stay useful when the application chooses and preserves the right material; a short question can still leave the model missing something you thought it knew.

Imagine asking, “Is that exception still in force?” You know which exception you mean. The model needs enough of the earlier conversation in its next request to know it as well. The words in the message box are only the visible beginning of the question.

Your short question has company

Behind that little message, the application may send its own instructions, earlier messages, document passages, descriptions of available tools and the results those tools returned. Depending on the product, earlier reasoning or summaries may come along, too. Message formatting also takes space. The context limit applies to this assembled request, not just the sentence you have typed.

Before the model works with the text, a tokenizer divides it into tokens. A token can represent a whole word, a piece of one, punctuation or another sequence of bytes. Each receives a vocabulary identifier, a number the model can work with. When the model generates identifiers for its answer, they are decoded back into text.

That is why counting words, or even characters, will not give you an exact token count. A familiar sentence, a stretch of code and a string of unusual service names can take different amounts of room despite looking much the same length. Mixed languages can change the count, too. The particular tokenizer decides where those pieces begin and end.

To see that count without guessing, take this small, invented certificate plan. One sentence says what to do, another says what must wait, and a background note repeats forty times. The repetition is deliberate: it makes the difference between space occupied and importance rather hard to miss.

The Python package tiktoken can count the supplied text with a named encoding. Use a disposable virtual environment for the example below. It fixes o200k_base so the counting method is reproducible, not because every model uses that encoding.

python3 -m venv .venv
. .venv/bin/activate
python -m pip install tiktoken

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
parts = {
    "task": "Summarize the proposed certificate rotation.",
    "constraint": "Do not rotate the legacy payment endpoint before its client update.",
    "notes": "The migration uses separate modern and legacy endpoints.\n" * 40,
}
for name, text in parts.items():
    print(name, len(enc.encode(text)))
print("joined text", len(enc.encode("\n".join(parts.values()))))

The program prints a count for each part and another for the joined text. Those counts need not add up as you might expect: joining text can change the token boundaries. Run it to see the actual counts; none are invented here.

It is also a count of these strings, not a complete bill for a chat request. It leaves out the provider's full message packaging and does not predict how images or audio are counted. For a real request, the selected interface's supported counting or usage facilities are the better place to check.

Look back at the one-line constraint beside those forty repeated notes. The tokenizer can count both. It cannot tell you that the short sentence is the reason the old payment endpoint must stay untouched.

The answer needs room, too

Suppose an interface gives a request and its answer a combined allowance of 8,000 tokens. You send 7,400. There are 600 left, even if you were hoping for a generous 2,000-token answer. Input and requested output would total 9,400, which is 1,400 beyond this hypothetical limit.

The arithmetic is simple; the product's response to it is something to check. An interface may reject the request or use its documented limiting or truncation behavior. Exact input and output limits vary by model and interface. An output limit can be restrictive even when the input fits, and some models also account for internal reasoning tokens. The current model specification and endpoint controls tell you which allowances apply.

A larger window can make room for more of your notes. That is useful. It still does not tell you whether the answer used the sentence about the old endpoint. One practical way to find out is to ask a question that depends on that sentence, then ask it again after changing the context. Now you are checking whether the important information survived, rather than simply admiring how much text fits.

A shorter plan can quietly become a different plan

Return to the long conversation. To keep working within the available space, an application may rewrite earlier material into a shorter summary. This is often called compaction. It can keep a conversation going, but the next request now receives the summary in place of some of the original wording.

Our certificate notes offer an easy place to start cutting: forty repetitions of the same background sentence. The condition about the old payment endpoint needs more care. “Rotate the payment certificates after preparation” is attractively brief, but it has lost the distinction between the modern and legacy endpoints. The legacy endpoint must wait for its client update, and a useful summary needs to say whether anyone knows that update has happened.

Saving a few tokens is a poor bargain if someone has to go back and recover the missing condition. Keep the applicable conditions while trimming repeated background, then read what the next request will actually contain. If a short summary cannot carry what matters reliably, a smaller request with the necessary source, or retrieval of the original passage again, can put it back in view.

Same facts, different shape

You can make the difference visible with this invented reference: “Modern endpoints may use the shared renewal job. The legacy payment endpoint must wait for its client update. This note does not say whether that update happened.”

For request A, ask: “Using only the reference, explain the renewal rule for modern and legacy endpoints in three sentences.” For B, leave the reference unchanged and ask for the same explanation in a two-row table. Keep the reference, both instructions and both answers together. An available model interface will let you compare actual responses; working on paper is also useful, as long as the expected comparison is labeled unexecuted.

The interesting change is the shape of the answer. Three sentences become two rows. The modern endpoints may still use the shared job, and the legacy one still needs an update whose completion is unknown. A neatly arranged table has no more reason than a paragraph to decide that the update happened.

Now replace the reference with a deliberately poor summary: “All payment endpoints can rotate after preparation.” Repeat request A. The condition is no longer in the material supplied to the model, so it cannot reliably reconstruct it. Asking for extra care cannot bring back a sentence the application left out.

With the prompts beside the answers, you can point to exactly what changed: first the requested presentation, then the information available. That distinction is useful whenever a conversation gets shortened. The next answer may look perfectly orderly even though it has been asked a different question.

What the smaller request really saves

There is a good reason to care about the count. Tokens affect cost and waiting time. Hosted services can charge differently for input, cached input and output; local inference consumes hardware time and memory. The selected service's current rates and cache rules matter. Repeating text does not, by itself, establish that it received a cache hit.

But the work does not end when a small request returns. If its answer loses the exception, retrieving the passage again, retrying and repairing the plan add to the cost of finishing. The token cost guide follows those costs beyond a single call.

Scroll back to that long conversation one more time. You do not have to send every repeated note forever. You do need the next request to carry the facts the next answer depends on, with enough room left for that answer. In our certificate plan, the sentence worth keeping is short. Its importance has very little to do with its token count.