Build an inference compatibility suite before changing the runtime

The application expects a timeout of 30, a number. The new model server returns "30", text. A person reads the same thirty seconds; the application rejects it. This hypothetical mismatch is why loading the model successfully is only the beginning of an inference migration.

An r/LocalLLaMA discussion asked how practitioners test the same open model across inference engines, quantizations, chat templates, and hardware. Replies converged on a useful operating idea: pin the exact stack, use a small fixed suite, and preserve raw outputs. The thread is a discovery source, not evidence that one runtime is universally compatible.

Save the request, raw reply and the application’s result together. They show whether a change preserved the thing the consumer needs, and give a failed test somewhere more useful to point than “the new model looks different.”

When the model computes an answer from input, it is performing inference. The runtime loads it and carries out that computation. A chat template arranges messages in the format the model expects; an API adapter packages the exchange for the application. Quantization uses lower-precision numbers to save resources and can change the calculation. Each sits between the saved request and the value the client finally receives.

This guide assumes you can send a saved request to a test endpoint and inspect the response. You do not need to train a model. Start with the exact response your application must consume, then work backward to the components that produced it.

Define compatibility at the application boundary

Start with the response your code must read. Does it have the fields, types, tool name and ending the client expects? A beautifully written reply can fail there. Comparing prose similarity would miss the integer turned into text.

Describe each interface in the terms its consumer needs. A tool call asks the application to run a named function with supplied arguments; test its name, argument types, allowed order, and a case where no tool applies. Structured output must fit a schema, the field-and-type rules the consumer expects, with a defined rejection path. Streaming delivers the response in pieces, so test event order, the ending, usage fields, cancellation, and partial failure. Context handling depends on the template, special tokens, truncation policy, and tested length. These are concrete things a serving change can preserve or break.

The OpenAI evaluation guidance makes a general point that transfers across providers: evaluations should be task-specific, logged, and run continuously instead of relying on impressions. For an open-model migration, the task is not “produce good prose.” It is “preserve the application contract closely enough for this workload.”

Pin the whole serving stack

Suppose we hand the rejected timeout to a colleague. “Same model, new server” leaves them several places to look. Start with which model actually ran: its repository and immutable revision or file hash. The tokenizer files and chat template tell them how our messages became model input. Quantization method and parameters, engine version, backend libraries, driver and accelerator describe the calculation. Sampling settings and seed behavior describe how output was selected; the API adapter describes how it reached the client. Each belongs in the record because each sits on the timeout’s route.

The colleague can now ask a sharper question: were the quotation marks around 30 already in the model’s output, or did they appear later? An adapter can change the reply on its way out. Earlier in the route, a template may have inserted role or reasoning markers that were not visible in our prompt; numerical backends and quantization may also have changed the computation. Even an omitted setting is not necessarily absent: if the runtime supplied a default, its resolved value belongs in the record. The saved pieces turn “looks different” into places we can inspect.

The vLLM OpenAI-compatible server documentation and the SGLang API documentation describe compatibility surfaces, but an interface label is not a guarantee for every model, template, extension, or client. Treat documentation as the starting contract and your suite as the evidence for the exact combination you run.

Build a small suite that fails usefully

For each saved test request, write what its consumer must be able to do with the result and keep the raw reply. The tests for a tool call, JSON object, stream ending and near-full conversation should ask different questions. Name the model, template, runtime and hardware beside each run.

Include a valid tool call, an ambiguous choice, recovery from malformed arguments, strict JSON, cancellation, a stop-sequence edge, nearly full context and a repeated conversation. Add sanitized incident cases when they reveal an actual fragile point. Each case should earn its place by exposing a difference that matters to the client.

Consider a hypothetical twelve-case suite for an internal incident-triage assistant. The reference runtime returns a tool call with an integer timeout; the candidate returns the same digits as a string. A person reading the chat may see no difference. A typed dispatcher does, and rejects the candidate. That one small failure tells you more about application compatibility than a high average similarity score.

This small synthetic assertion checks the integer-versus-string boundary after tool arguments have been decoded. It does not call either runtime:

reference = {"timeout": 30}
candidate = {"timeout": "30"}

def valid_timeout(arguments):
    return type(arguments.get("timeout")) is int

assert valid_timeout(reference)
assert not valid_timeout(candidate)

Both assertions pass because the first value has the required integer type and the second does not. Both objects can be valid JSON: syntax alone does not enforce the application’s integer requirement. Python also treats booleans as a kind of integer for some checks, so this deliberately uses an exact type comparison rather than accepting True as a timeout. Keep the original response beside the assertion result. It is how you determine whether the model generated a string, an adapter converted the value, or another serialization step changed it.

Be exact where the interface is exact. JSON types, event order, stop state, and tool names can need precise assertions. Natural-language explanations may instead need required facts, prohibited claims, or a review rubric. Insisting on identical bytes from a behavior that was never deterministic would produce failures your application does not actually care about.

Change one factor and keep the baseline

We want to know whether changing the runtime changed our timeout. If the model revision, template, quantization, prompts, sampling settings or hardware changed too, the comparison cannot answer that question cleanly. Keeping them steady lets us examine the runtime change. To examine quantization instead, keep the engine and template steady. A migration may force several changes together; then we are comparing bundles, not proving which one component caused the difference.

For the trial, both servers start cleanly and warm under the same policy. The saved request goes to each, and the record keeps response bytes and available token identifiers alongside timing, memory, warnings and server errors. Suppose the candidate replies sooner but still wraps 30 in quotation marks. Its speed has not made that tool call usable. If the type is correct but the reply is too slow, we have a different problem. Checking application use before performance keeps those two decisions separate.

Model behavior can also change between snapshots. The OpenAI API reference recommends pinned model versions and evaluations for applications that need consistent behavior. Open models give operators more control over artifacts, but the same discipline applies: a floating tag is not a reproducible baseline.

Classify differences before choosing a winner

Follow the timeout value through its journey. If the raw model output already contains a string, inspect generation, the template, and the structured-output path. If the raw output is an integer but the client gets a string, inspect the adapter. An expected wording variation belongs in neither category. Classify a difference only after the request and response show where it appeared.

If the team cannot tell whether a difference is harmless wording or a broken rule in an action-bearing response, settle that question before rollout. A mostly plausible transcript cannot explain an unresolved timeout type or tool name.

Keep the comparison matrix with immutable component IDs and its review date. Each row describes one setup under one suite. Changing the model, template, backend, driver, hardware or client creates another setup to compare; the earlier replies remain useful evidence when a regression appears.

The related model-and-harness benchmark guide explains why changing orchestration and model together limits attribution. The shadow-test migration guide shows how to compare candidates without giving them production authority. Keep those comparisons beside the raw compatibility results. If the integer timeout becomes a string, the deployment decision can then name the exact broken contract and the version to restore.