On this page5 sections
The first sentence appears almost immediately. You start reading, reach the part you need explained, and wait. The stream has stalled. A timer that stopped at the opening would remember the pleasant part of this exchange and miss everything that followed.
Measure both moments at the client: first answer text and completed response. Keep completed, rejected and interrupted requests beside those times under the same offered load. Server token speed helps explain capacity; client outcomes show who actually received the answer.
Imagine a responder reading a streamed summary of sanitized logs. An early opening is welcome, but the full account is needed to judge it. Starting sooner while abandoning more summaries would improve the greeting and worsen the job.
Follow the first readable text, the stream’s finish and the answer’s acceptance as useful output. Then compare configurations receiving the same incoming work. A precise clock cannot rescue a test that quietly stopped submitting its difficult requests.
Start the clock where the reader starts waiting
Start at submission. Response headers arrive, perhaps followed by an event that names the assistant’s role. Neither gives the responder a word to read. The first nonempty answer text does. Time to first text runs to that event, even if separate reasoning output arrived earlier without supplying the answer.
An engine’s time to first token, or TTFT, can cover another interval. Its clock observes the serving path; the client also waits for connections, proxies and delivery. Provider TTFT need not equal client first-text time. Nor is received text necessarily visible yet. Instrument the real interface separately to measure when it renders for the reader.
The pause in the summary also needs careful naming. Server-Sent Events, or SSE, deliver pieces that may contain phrases, metadata or errors. One event can carry several tokens. A client can time those deliveries, but not reconstruct when each token inside a bundle was generated. Smooth arrival and smooth generation are different observations.
When the remaining text arrives, look for a normal finish and the expected stream termination. An output limit may cut the answer short. An application consuming structured output also needs it to parse successfully and contain the required fields before its own work is complete.
The vLLM metrics design distinguishes gaps between output events from request-level time per output token. The latter divides post-TTFT time by the number of output tokens minus one. Bundled tokens and different weighting can make these measures disagree. Keep definitions and request populations with the numbers so delivery intervals do not quietly become generation intervals.

Inspect what the client probe actually counts
The standard-library probe below sends a synthetic request to an OpenAI-compatible endpoint. It skips role-only events, times nonempty answer text and requires both a normal stop and [DONE]. HTTP 429 and 503 count as rejections; malformed or interrupted streams remain unsuccessful.
This is a diagnostic, not a load generator or a quality test. Its handling was checked with a local synthetic SSE fixture, not a serving benchmark. Endpoints with different termination rules need a deliberate adaptation.
import http.client
import json
import os
import socket
import sys
import time
import urllib.error
import urllib.request
def events(response):
data = []
for raw in response:
line = raw.decode("utf-8").rstrip("\r\n")
if line == "":
if data:
yield "\n".join(data)
data = []
elif line.startswith("data:"):
data.append(line[5:].lstrip(" "))
def probe(url, model):
payload = {
"model": model, "stream": True, "temperature": 0,
"max_tokens": 128,
"messages": [{"role": "user", "content":
"Explain what a Kubernetes readiness probe checks in three sentences."}],
}
headers = {"Content-Type": "application/json", "Accept": "text/event-stream"}
if os.environ.get("LLM_API_KEY"):
headers["Authorization"] = "Bearer " + os.environ["LLM_API_KEY"]
request = urllib.request.Request(url, json.dumps(payload).encode(), headers)
started = time.perf_counter()
first = finished = None
reason = None
text_events = chars = 0
outcome = "interrupted"
try:
with urllib.request.urlopen(request, timeout=30) as response:
if "text/event-stream" not in response.headers.get("Content-Type", ""):
raise ValueError("expected SSE")
for event in events(response):
now = time.perf_counter()
if event == "[DONE]":
outcome = "completed" if reason == "stop" and first is not None else "noneligible_finish"
if outcome == "completed":
finished = now
break
item = json.loads(event)
if "error" in item:
outcome = "api_error"
break
for choice in item.get("choices", []):
if choice.get("index", 0) != 0:
continue
content = choice.get("delta", {}).get("content")
if isinstance(content, str) and content:
first = now if first is None else first
text_events += 1
chars += len(content)
if choice.get("finish_reason") is not None:
reason = choice["finish_reason"]
except urllib.error.HTTPError as error:
outcome = "rejected" if error.code in (429, 503) else "http_error"
except (TimeoutError, socket.timeout):
outcome = "timeout"
except (urllib.error.URLError, http.client.HTTPException,
OSError, ValueError, TypeError, AttributeError):
outcome = "client_or_protocol_error"
return {
"outcome": outcome, "finish_reason": reason,
"first_text_seconds": None if first is None else first - started,
"completion_seconds": finished - started if outcome == "completed" else None,
"observation_seconds": time.perf_counter() - started,
"text_events": text_events, "output_characters": chars,
}
if __name__ == "__main__":
print(json.dumps(probe(sys.argv[1], sys.argv[2]), sort_keys=True))Save it as stream_probe.py, establish the real endpoint and model ID, and use that endpoint’s supported template. Supply LLM_API_KEY through the environment if needed. The llama.cpp reference explains its endpoint configuration.
This example invocation assumes port 8000. Replace the URL and model placeholder with your configured values:
python stream_probe.py http://127.0.0.1:8000/v1/chat/completions YOUR_MODEL_IDThe probe does not print the key or retain the answer text.
Consider an invented result: first_text_seconds is 0.4, completion_seconds is null and outcome is interrupted. Answer text reached the client after four tenths of a second, but the required normal finish never arrived. That is our stalled summary in a few fields, not a quickly completed answer. It is not an observed probe run.
If first text is missing as well, no answer text arrived at this client. The other counters describe deliveries: text_events counts events and output_characters counts characters, not tokens. Neither tells you whether the contents helped the responder; the probe keeps no answer and makes no quality judgment.
The 30-second socket timeout limits blocking network operations rather than the whole request’s lifetime, as the Python documentation explains. A continuing trickle can keep the stream alive longer. Load tests need the actual application deadline, held steady between configurations.
Keep new work arriving during the comparison
One person waits for a summary before asking another question. When the service slows, that person sends less work. A test client that waits the same way also eases the load at the moment the server struggles. That can represent a fixed interactive group, but not a service where new requests keep arriving regardless.
Choose which situation the test represents. The vLLM benchmark reference distinguishes a requested arrival rate from a concurrency cap. During long requests, a cap can reduce actual submissions. Report intended and actual arrivals together, or the generator’s restraint may look like a server speedup.
Keep sanitized prompts, model, quantization, sampling and output limits the same while changing the serving variable being tested. Record actual lengths when available, since a shared output cap does not give every answer the same length. Separate small summaries from large evidence bundles so an improvement for one group cannot hide a longer wait for the other.
A second reading of the same log packet may reuse its prefix and take less work. That suits repeated runbook context, but may overstate capacity for fresh evidence. Make warmup and cache conditions explicit. Match them or report them separately instead of crediting a warm replay as an improvement for fresh requests.
The tuning guide compares latency, finished work and rejection under equal offered demand. For this responder, add the two moments the screen makes tangible: when the answer starts and when it finishes.
A faster first sentence can reach fewer people
Here is invented accounting, not benchmark data. Both configurations receive the same 100 requests, arrival pattern and deadlines. Count each request once:
| Outcome | Configuration A | Configuration B |
|---|---|---|
| Offered requests | 100 | 100 |
| Normally completed responses | 80 | 60 |
| Rejected before an answer | 10 | 30 |
| Interrupted or timed out | 10 | 10 |
| Completed responses satisfying timing and application checks | 72 | 54 |
| Median first-text time among normal completions | 0.9 seconds | 0.4 seconds |
B starts its normally completed answers sooner, but fewer requests reach that finish and fewer satisfy timing and application checks. Useful completion is 54%, compared with A’s 72%. The first-text median describes normal completions only; by itself, it hides the refused and lost work.
Someone rejected by B never experiences its faster opening. The median can be correct for its selected group and still say little about everyone who asked. Keep all 100 requests visible without assuming that every protective rejection was a bad policy decision.
Useful completion also needs an application judgment. This summary must address its evidence and preserve uncertainty. A terminal event establishes delivery, not truth. Likewise, latency goodput counts work meeting timing conditions; it does not establish answer quality without a separate evaluation.
Rejecting work can protect a constrained dependency from collapse. Compare that protection with the overloaded baseline, include refused work and later retries or deferral, and judge the accepted outcomes. It is bounded protection, not an unconditional speed gain. Retries themselves add offered work.
Use the server record to explain the wait
Now suppose first text is prompt but the completed answer is late. The client record shows the long interval; server measurements can help distinguish waiting, generation and delivery delay. vLLM production metrics include queue, TTFT and end-to-end observations. Match the installed version and actual metric names before lining them up with the client result.
Enabled llama.cpp metrics also expose generated-token counts and throughput. A rising counter establishes that the engine did work, not that this responder received the rest of the summary. Fleet token totals can rise while particular people lose responses. Keep the engine and client records because they answer different questions.
Across replicas, Prometheus explains how histograms can be aggregated, whereas percentiles cannot simply be averaged. Its rate guidance calculates rates before summing so counter resets remain accounted for. The histogram guide develops the queries once the request population and completion rules are clear.
The quick first sentence now has the rest of its story beside it: whether the answer finished, whether it was useful and how larger requests fared under the same workload. Keep the timing gain when it does not conceal unacceptable lost completions or large-request regressions. Save the prompts, arrivals, deadlines and outcomes so the next engineer can follow the whole answer, not rediscover the part the first timer left out.
Source context
Source links appear within this article. Read them alongside the author’s analysis and evaluate the guidance against your environment.
Report an error or outdated detail