Build an inference compatibility suite before changing the runtime

A model that loads successfully has passed a transport check, not a compatibility test. The weights may fit in memory while the serving path still changes tool calls, structured output, stop behavior, streaming, or long-context handling. Those differences become production failures when an application assumes that changing the runtime is invisible.

A recent r/LocalLLaMA discussion asked how practitioners test the same open model across inference engines, quantizations, chat templates, and hardware. Replies converged on a useful operating idea: pin the exact stack, use a small fixed suite, and preserve raw outputs. The thread is a discovery source, not evidence that one runtime is universally compatible.

ChatGPT is most useful in an SRE workflow when the output has a factual boundary the responder can inspect. The same principle applies to an open-model serving change. Pin every movable part, change one, and preserve raw outputs beside pass or fail.

Define compatibility at the application boundary

Start with the behaviors that downstream code consumes. A chat response that looks reasonable to a person can still violate a tool schema, omit a required field, split a streaming event incorrectly, or ignore a stop condition. Text similarity alone does not establish compatibility.

Write a short contract for each interface. For tool calling, name the available tools, required argument types, permitted tool order, and behavior when no tool applies. For structured output, validate the exact schema and rejection path. For streaming, check event order, final usage fields, cancellation, and partial failure. For context handling, record the template, special tokens, truncation policy, and maximum tested length.

The OpenAI evaluation guidance makes a general point that transfers across providers: evaluations should be task-specific, logged, and run continuously instead of relying on impressions. For an open-model migration, the task is not “produce good prose.” It is “preserve the application contract closely enough for this workload.”

Pin the whole serving stack

Record the model artifact by repository and immutable revision or file hash. Record the tokenizer files, chat template, quantization method and parameters, inference engine version, backend libraries, driver, accelerator, sampling settings, seed behavior, and API adapter. If a parameter is omitted and a runtime supplies a default, record the resolved value.

This is not administrative detail. A template can insert a reasoning marker or tool boundary that another engine interprets differently. Quantization can alter close token probabilities. An attention backend can take a different numerical path. A server may translate an OpenAI-compatible field but implement a different default. The deployed object is the complete stack, not only the model name.

The vLLM OpenAI-compatible server documentation and the SGLang API documentation describe compatibility surfaces, but an interface label is not a guarantee for every model, template, extension, or client. Treat documentation as the starting contract and your suite as the evidence for the exact combination you run.

Build a small suite that fails usefully

The suite runs fixed scenarios for tool calling, structured output, streaming, stop behavior, and long-context handling against an exact model artifact, template, runtime, and hardware target. Each case needs a request fixture, an assertion at the consumer boundary, and saved raw output sufficient to explain a failure.

Prefer a dozen discriminating cases over hundreds of vague prompts. Include one valid tool call, one ambiguous tool choice, one malformed-argument recovery case, one strict JSON schema, one stream cancellation, one stop-sequence boundary, one request near the context limit, and one conversation that exercises the template repeatedly. Add cases from real incidents after removing private data.

Consider a hypothetical twelve-case suite for an internal incident-triage assistant. The reference runtime emits a tool call with an integer timeout. A candidate runtime emits the same value as a string. Both responses look correct in a chat transcript, but only the reference passes the typed dispatcher. That one failure is more useful than a broad similarity score because it names the broken boundary and its consequence.

Keep exact assertions where the protocol requires them and semantic assertions where harmless variation is expected. JSON types, event order, stop state, and tool names can be exact. Natural-language explanations may be evaluated with required facts, prohibited claims, or a reviewer rubric. Do not force byte equality onto a behavior that was never deterministic.

Change one factor and keep the baseline

When evaluating a new runtime, hold the model revision, template, quantization, prompts, sampling settings, and hardware steady. When evaluating a new quantization, hold the engine and template steady. If operational constraints force several changes together, label the result as a bundle comparison and avoid attributing a difference to one component.

Run the baseline and candidate from clean starts. Warm them according to the same policy. Capture request and response bytes, token identifiers when available, timing, memory, warnings, and server errors. Separate correctness from performance. A faster runtime that breaks a tool contract has not passed, while a compatible runtime that misses the latency objective needs a performance decision rather than a quality claim.

Model behavior can also change between snapshots. The OpenAI API reference recommends pinned model versions and evaluations for applications that need consistent behavior. Open models give operators more control over artifacts, but the same discipline applies: a floating tag is not a reproducible baseline.

Classify differences before choosing a winner

Every failed comparison should enter one of four buckets: expected variation, client incompatibility, runtime defect, or application assumption. An expected wording change may need no action. A changed JSON type can require an adapter, a runtime fix, or rejection of the candidate. A prompt that depended on an undocumented template behavior may reveal an application assumption worth removing.

Stop the rollout when the suite cannot distinguish an expected output difference from a broken protocol contract. That boundary prevents teams from approving a migration because most examples “look fine” while one unexplained difference remains in an action-bearing path.

Publish the matrix with immutable component identifiers and a review date. A compatibility row is evidence about one combination under one suite, not a permanent endorsement of a model or engine. Rerun it when the model, template, runtime, backend, driver, hardware, or client changes. Archive the evidence rather than silently replacing old results.

The related model-and-harness benchmark guide explains why changing orchestration and model together limits attribution. The shadow-test migration guide shows how to compare candidates without giving them production authority. Together, they turn a runtime swap from an anecdotal demo into a reversible engineering decision.