Tokens and context windows: what your model actually receives

A long conversation can stop being useful before it reaches a model's advertised limit. The next request may include instructions, earlier messages, retrieved documents and tool results, all competing for space. Understanding tokens and context windows helps you decide what to send, what to shorten and what must remain available for the answer to be safe and useful.

A token is a unit in the model's input or output vocabulary. Depending on the tokenizer and text, it may represent a word, part of a word, punctuation or another byte sequence. Tokens are not words, and a character estimate is especially unreliable for code, identifiers and mixed languages. The context window limits the sequence the model can work with during a request; exact input and output limits depend on the model and serving interface.

See the request that will be sent

Start by inspecting the assembled request rather than the text in the message box. A chat application may add instructions, message formatting and tool definitions. Retrieval adds passages. An agent adds tool results and sometimes earlier reasoning or summaries, depending on the product. A short new question can therefore produce a large request.

For a local counting exercise, use Python in a disposable virtual environment and install tiktoken. It can count text with a named encoding. The example uses a specific encoding to make the exercise reproducible, not to claim that every model uses it.

python3 -m venv .venv
. .venv/bin/activate
python -m pip install tiktoken

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
parts = {
    "task": "Summarize the proposed certificate rotation.",
    "constraint": "Do not rotate the legacy payment endpoint before its client update.",
    "notes": "The migration uses separate modern and legacy endpoints.\n" * 40,
}
for name, text in parts.items():
    print(name, len(enc.encode(text)))
print("joined text", len(enc.encode("\n".join(parts.values()))))

This counts the supplied text. It does not count a provider's complete chat envelope, predict image or audio accounting, or certify that a request will fit a particular API. Use the serving interface's supported token-counting or usage facilities for that final check. Encoding a joined string can also differ from adding separate counts because boundaries affect tokenization.

Reserve space for the answer

Imagine a hypothetical serving configuration with a combined 8,000-token allowance. If the assembled input occupies 7,400 tokens, requesting another 2,000 output tokens does not create extra capacity. Some interfaces reject the request; others apply their documented limits or truncation behavior. Verify that behavior rather than treating silent shortening as a harmless convenience.

Output limits are separate controls as well. A model may accept a large input but permit a smaller maximum output. Some models account for additional internal reasoning tokens. Read the chosen model's current specification and the endpoint's controls before turning a window size into a usable-input calculation.

A larger window lets you supply more material, but it does not prove that every relevant detail will influence the answer correctly. Keep a test question whose correct answer depends on an easily lost condition. For the illustrative certificate rotation, the exception for the legacy endpoint is more consequential than another paragraph describing the overall migration.

Shorten with a preservation test

Compaction rewrites earlier material into a shorter representation. It can make continued work practical, but it changes the evidence available to the next request. Shorten repetitive background first, retain the applicable constraints and inspect what the next assembled request actually contains.

In the certificate example, compare the original notes with a proposed summary. A summary that says “rotate the payment certificates after preparation” has lost the condition that distinguishes the legacy endpoint. Reducing the token count would be a technical success and a task failure. A useful summary keeps the endpoint distinction, the prerequisite client update and any unresolved question about whether that update has happened.

When the condition cannot be preserved reliably, start a new bounded request with the necessary source material or retrieve the relevant passage again. Continuing a shortened conversation is not mandatory. The right choice depends on whether the information required for the next decision remains available.

Cost follows the whole task

Tokens also affect cost and latency. Hosted services can price input, cached input and output differently, while local inference consumes hardware time and memory. Current rates and cache rules belong to the service you choose. Do not apply a remembered price to another model or assume a cache hit from repeated wording.

A smaller call is not necessarily a cheaper completed task if it creates retries or loses a condition that must be recovered. Count the calls needed for the useful result, including retrieval, correction and verification. The token cost guide develops that accounting; this lesson's job is to make the request and its preservation choices visible before you optimize the bill.