Bit-exact decoding turns inference speedups into testable changes

A speculative decoder cannot support a model-quality comparison when the inference engine changes outputs on its own. Speed and equivalence are separate acceptance questions. If the optimized path selects different tokens, the benchmark is no longer measuring only faster execution of the same serving contract.

A recent r/LocalLLaMA post described work on bit-exact speculative decoding for Qwen3.8-27B in SGLang and compared behavior with vLLM. The author reported token and log-probability differences across configurations, then reported exact agreement after patches in a limited prompt set. Those measurements are author-reported and have not been independently reproduced here.

The speedup and the risk disposition are separate decisions. Measure throughput and output equivalence as separate acceptance criteria. Passing one cannot compensate for an unexplained failure in the other.

Separate the decoding method from the served result

Speculative decoding uses a cheaper draft process to propose one or more tokens, then asks the target model to verify them. Accepted proposals reduce the number of expensive target-model steps. The optimization is attractive because it can improve throughput without changing the target model weights.

A draft model proposes tokens, and the target model verifies them before acceptance; implementation details in the verification path can still change token selection or log probabilities. Floating-point reductions, attention kernels, probability normalization, sampling order, and cache handling can all matter near a decision boundary. “Same model” does not automatically mean “same decoding result.”

The distinction matters beyond reproducibility. If a team compares quantizations, prompts, or model revisions while the decoder itself introduces variation, it can misattribute a quality change. The benchmark should first establish what the serving path preserves.

Define bit-exact before reporting it

“Bit-exact” can refer to different observables. A strict claim may require identical token identifiers, token order, stop condition, and log probabilities represented with the same precision. A narrower claim may cover only greedy token output for a fixed request. State the definition before presenting a pass rate.

Pin the model artifact, tokenizer, chat template, runtime commit, backend libraries, kernel selection, driver, accelerator, batch shape, context length, sampling parameters, seed, and numerical precision. Record whether the reference and candidate use the same attention backend. If they do not, the test is a comparison of two serving bundles.

The thread links an SGLang pull request and a related DeepGEMM pull request. Those primary engineering records are useful because they expose the proposed changes and review discussion. They do not, by themselves, establish universal exactness across devices, models, loads, or future commits.

Build a reference corpus that exposes boundaries

Begin with deterministic decoding so random sampling does not hide implementation differences. Include short and long prompts, repeated prefixes, tool schemas, stop sequences, batched requests, and context lengths that cross cache or kernel thresholds. Preserve request fixtures and raw response records.

Consider a hypothetical one-hundred-prompt benchmark that compares a reference decoder with a speculative path. Ninety-nine prompts return identical tokens. One long-context prompt diverges at token 1,842 after a backend switch. Reporting “99 percent exact” conceals the operational clue. Reporting the request shape, divergence point, backend, and log-probability delta gives an engineer a path to reproduce the boundary.

Compare token identifiers at every position and stop as soon as they diverge. Compare target-model log probabilities before sampling when the runtime exposes them. Capture accepted draft length, rejection position, attention backend, batch state, and cache occupancy. A mismatch should produce an artifact, not only a red test.

Once deterministic equivalence is understood, test the sampling modes the service actually uses. Exact replay may require identical random-number generation and operation order, which can be unrealistic across implementations. If exact replay is unavailable, define a weaker service-level contract and keep conclusions within it.

Measure performance without weakening the gate

Run throughput and latency trials separately from correctness comparison. Measure time to first token, inter-token latency, requests per second, output tokens per second, memory use, and error rate under realistic concurrency. Include the draft model's memory and compute cost. A headline speedup without its workload shape cannot guide capacity planning.

Use the same request distribution and output limits for reference and candidate. Warm both paths consistently. Randomize run order when infrastructure load can drift. Report medians and tails, not only the best run. Preserve failed requests in the denominator.

Do not tune away a correctness failure. If changing a kernel, batch size, or capacity parameter restores equivalence, treat that configuration as a new candidate and rerun both gates. The safe conclusion is about the tested bundle, not speculative decoding in general.

Choose the claim that the evidence supports

If exact token and log-probability replay passes across the bounded corpus, the team can claim equivalence for those observables, versions, and conditions. It still needs application-level tests for streaming, tool calls, timeouts, and cancellation. Mathematical equivalence at one internal boundary does not prove every service behavior.

If exact replay is unavailable, keep claims at the observed service level and do not attribute quality changes to the model or quantization alone. A team may accept bounded semantic variation for a conversational product. It should not use that result to compare model quality unless the serving variation is controlled or measured.

The inference compatibility suite provides the broader protocol tests around this decoder-level check. The model-and-harness benchmark guide provides the attribution rule: when several system factors move, describe the system result.

A useful release record therefore contains two verdicts. First, did the speculative path preserve the defined output contract? Second, did it improve the service objective under the intended load? Bit-exact testing makes the first verdict inspectable. A separate capacity trial makes the second defensible.