On this page5 sections
The saved request finishes sooner after an optimization. Then, partway through the reply, a token changes. Is this the same behavior delivered faster, or just an answer that reads similarly? Speculative decoding makes the distinction important: a smaller draft process proposes several tokens and a target model checks them together, potentially saving target-model steps. Exact replay and preserving the promised sampling distribution are different requirements.
A LocalLLaMA report describes speculative decoding for Qwen3.8-27B in SGLang and comparisons with vLLM. The author observed token and log-probability differences between configurations, then reported agreement after patches on a limited set of prompts. Those results have not been independently reproduced here.
Two questions deserve separate answers: does the candidate preserve the required output, and does it improve speed or capacity? Faster generation does not explain an unexpected change in the reply.
The same weights can still produce a different next token
Picture two nearly tied next-token scores. Rounding nudges one score above the other on one serving path and below it on another. Identical model weights can then lead to different tokens. Attention, normalization and cache handling contribute to those scores; changing their arithmetic can change the served answer.
With limited precision, even addition order matters. Add a large positive number to its large negative counterpart first, and a small remaining number survives. Add the small number to the large one first, and rounding can lose it. GPUs can rearrange additions for speed. Near a token-choice boundary, that difference can change the next token and everything following it. Log probabilities express likelihood on a logarithmic scale. They may expose a difference before the text changes.
If quantization changes alongside decoding, a different answer has two possible explanations. Establish what the decoder preserves before attributing later quality changes to quantization. The aim is a comparison we can interpret, not identical prose from every system.
Say exactly what “bit-exact” compares
Put the old and new replies together and specify what must match. Strict replay may require token IDs, their order, stopping and log probabilities at matching precision. A narrower test may require only the same tokens from greedy decoding, which chooses the highest-scoring next token at each step rather than sampling, on fixed input. A “bit-exact” pass needs to name its observables, or the percentage conceals what was compared.
Another engineer first needs to know whether both paths received the same input: save the model artifact, tokenizer and prompt template. To reproduce the calculation, also retain runtime commit, libraries, attention backend, kernels, driver, hardware, batch shape, context, sampling, seed and precision. Otherwise they may investigate a decoder change while unknowingly running different arithmetic.
The SGLang change addresses the arithmetic in GPU code. Its Triton implementation groups small BF16 matrix products into a different tile, a block of work assigned together. Inputs use reduced-precision BF16; each output retains one more precise FP32 running sum. Along the shared matrix dimension K, addition order stays the same, with rounding once at the end. Work is regrouped without changing those arithmetic steps, letting speed be evaluated separately.
The author reports token and log-probability matches on bounded greedy tests using one RTX PRO 6000 and tensor parallelism one. The model is not split across multiple devices in that reported setup. That describes one setup, not every device. Speculation figures also use a development tree with additional patches. Another device or software bundle needs its own test rather than inheriting the reported match.
The separate DeepGEMM fix repairs a concurrency hazard: a producer must not reuse a shared-memory stage until its consumers finish. The fix orders those consumers before releasing the stage for reuse. It is not the tile-selection change above. A repeatable kernel answer still does not establish serving throughput.
Keep the prompt that exposes the boundary
Start with deterministic decoding so sampling noise does not obscure implementation differences. Retain short and long prompts, repeated prefixes, tool schemas, stops, batches and lengths crossing cache or kernel thresholds, together with raw responses.
Imagine a hypothetical hundred-prompt trial with ninety-nine identical results. The long outlier diverges at token 1,842 after a backend switch. Its shape, the switch and the first differing token give another engineer a reproducer. The headline percentage should not bury that useful case.
Compare token IDs position by position. Where available, compare target log probabilities before sampling. Save accepted draft length, rejection position, backend, batch and cache state with the input and both replies.
Next examine the sampling modes the service actually uses. Exact replay can depend on random-number generation and operation order that implementations handle differently. If the service accepts a weaker contract, state it and keep the conclusion within it.
Measure the system that actually serves the reply
Even after the saved request matches, concurrent arrivals can change the picture. Readers notice first-token delay and pauses between subsequent tokens. Operators also need requests and output tokens per second, memory and errors at realistic concurrency. Include the draft process’s compute and memory; excluding them prices a smaller system than the one serving the reply.
Keep the long request in the workload, even when short ones show a better speedup. Use the same request distributions and output limits, with consistent warming. Vary run order when load can drift, so one path does not consistently receive easier conditions. Report middle and tail behavior, not only the best run, and retain failed requests in the population.
Suppose a new batch setting fixes the mismatch but reduces throughput. That new configuration needs correctness and performance tests again. Combining the old setup’s speed with the new setup’s matching output would describe a result neither produced.
Attach the claim to the tested configuration
An exactness pass supports the checked observables under those versions and conditions. Streaming, tools, timeout and cancellation still need service-level checks. Internal equivalence cannot certify every interface.
Without exact replay, describe the service behavior observed rather than assigning differences solely to the model or quantization. A conversation product may accept variation; a model-quality comparison must still control or measure serving variation.
The compatibility suite addresses surrounding protocol behavior. The model-and-harness guide examines attribution when multiple system factors change.
Return to the request that diverged at token 1,842. The selected configuration must either preserve the stated behavior or operate under a deliberately chosen different contract. Then its speed describes a whole candidate the team can actually judge.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- r/LocalLLaMA postwww.reddit.com
- SGLang changegithub.com
- DeepGEMM fixgithub.com
