Bit-exact decoding turns inference speedups into testable changes

Imagine replaying a saved request after a serving optimization. The answer arrives sooner, but somewhere in the reply a different token appears. Did the optimization preserve the behavior you needed, or merely produce something that reads much the same? Speculative decoding makes this question worth asking: a smaller draft process proposes several tokens, and a target model checks them together. Accepting those proposals can save target-model steps. Whether the result must replay exactly or preserve the distribution promised by sampling is a separate choice.

A LocalLLaMA report describes Qwen3.8-27B speculative decoding in SGLang and comparisons with vLLM. Its author observed token and log-probability differences across configurations, then reported agreement after patches on a limited prompt set. These results have not been independently reproduced here.

Keep two decisions separate: does the candidate preserve the required output, and does it improve speed or capacity? Faster generation cannot resolve an unexplained output change.

Separate the decoding method from the served result

Suppose the next token is chosen from two almost equal scores. Rounding makes one score a little larger in one serving path and a little smaller in another. The paths can then choose different tokens even though they loaded the same model weights. Attention calculations, normalization and cache handling all feed those scores; changing their arithmetic can change the served answer.

Consider adding a large positive number, a large negative number and a small number with limited precision. Cancel the large pair first and the small number survives; combine the small number with the large one first and rounding can erase it. GPU calculations can rearrange such additions to run faster. Near a token-choice boundary, the resulting difference can change the next token and everything after it. Log probabilities express likelihood on a logarithmic scale and can expose a difference before the text changes.

If quantization and decoding both change, a different answer has two possible explanations. Establish the decoder's preservation behavior before assigning later quality differences to quantization. The requirement is an interpretable comparison, not identical prose from every system.

Define bit-exact before reporting it

Put the saved reply beside the new one. If the requirement is strict replay, matching prose may not be enough: the token IDs, their order, stopping and log probabilities at matching precision may all need to agree. A narrower test might ask only for the same greedy tokens on fixed input. “Bit-exact” needs to say which of these things was compared; otherwise a pass rate hides what passed.

Now imagine handing the differing replies to another engineer. The first question is whether the two paths received the same input, so the model artifact, tokenizer and prompt template belong with the saved request. The next is whether they calculated it the same way. That takes the runtime commit, libraries, attention backend, kernels, driver and hardware, plus batch shape, context, sampling settings, seed and precision. These are not extra decorations on the benchmark: without them, the engineer could be investigating a decoder change while unknowingly reproducing different arithmetic.

The SGLang change takes that arithmetic concern into the GPU code. It tries to make the work faster without rearranging the additions that can change our next token. Its Triton code groups small BF16 matrix products into a different tile, meaning a block of work assigned together. The input numbers use reduced-precision BF16, while each output keeps one more precise FP32 running sum. Along the shared matrix dimension, called K, the order stays the same; rounding happens once at the end. The work is grouped differently, but those arithmetic steps are preserved. We can therefore ask separately whether this regrouping is faster.

The author reports token and log-probability matches on bounded greedy tests on one RTX PRO 6000 with tensor parallelism one. That is a result for a particular setup, not an answer for the different device we might have beside us. The speculation figures also use a development tree with additional patches. Carrying the result to another device or software bundle means testing that combination, not borrowing the old match.

A separate DeepGEMM fix addresses another way the calculation can go wrong: a producer must not reuse a shared-memory stage before its consumers are finished. The fix orders those consumers before release for reuse. That is a concurrency hazard, not the tile-selection change above. Showing that a kernel repeats its answer still leaves the serving-throughput question unanswered.

Build a reference corpus that exposes boundaries

Begin with deterministic decoding to expose implementation differences without sampling noise. Keep fixtures for short and long prompts, repeated prefixes, tool schemas, stops, batches and lengths crossing cache or kernel thresholds, plus their raw responses.

Imagine ninety-nine identical results in a hypothetical hundred-prompt trial, with one long request diverging at token 1,842 after a backend switch. The outlier is useful work: its shape, switch and first divergence provide a reproducer. An impressive overall percentage should not bury them.

Compare IDs position by position and locate the first divergence. Where exposed, compare target log probabilities before sampling. Save accepted draft length, rejection position, backend, batch and cache state with the failing input and both responses.

Then test the actual sampling modes. Exact replay can depend on random-number generation and operation order that differ across implementations. If the service accepts a weaker contract, state it and keep conclusions there.

Measure performance without weakening the gate

Suppose the corrected path now preserves the saved request. We still need to see how it behaves when requests arrive together. A reader notices the wait for the first token and the pauses between later tokens; an operator also needs requests and output tokens per second, memory use and errors at realistic concurrency. The draft process consumes compute and memory too. Leaving it out would make the optimization look cheaper than the system that actually serves the replies.

Our long request should not disappear from the workload just because short ones produce a nicer speedup. Both paths need the same request distributions and output limits, with consistent warming. If load drifts while the trial runs, varying the run order helps avoid giving one path the easier conditions. Middle and tail results show more than the best run, and failed requests still belong to the tested population.

Perhaps changing the batch setting repairs the mismatch but costs some throughput. That is the candidate we now have: a new kernel or batch configuration needs both correctness and performance tests again. The old setup’s speed and the new setup’s matching answer cannot be combined into one result that neither setup produced.

Choose the claim that the evidence supports

A passing exactness test supports those observables under those versions and conditions. The service still needs streaming, tools, timeout and cancellation checks; an internal equivalence result cannot certify every interface behavior.

If exact replay is unavailable, describe observed service behavior without assigning differences to the model or quantization alone. A conversation product may accept variation, but a model-quality comparison must control or measure that serving variation.

The compatibility suite covers protocol behavior around this check, while the model-and-harness guide explains attribution when several system factors change.

Return to the request that diverged at token 1,842. Either the chosen configuration now preserves the stated behavior or the team has deliberately chosen another contract. Only then does the speed result describe a candidate you can judge as a whole.