Opens in a new tab
Guide

Test the reply your application needs before changing inference runtimes

In brief

A timeout returned as text can look fine in chat and fail in code. Build a small compatibility suite that preserves the request, reply and consumer result.

6 min read

Is this the right starting point?
Before you read
Practical API testing and model-serving experience, including structured outputs and deployment changes.
What you will take away
Build a pinned compatibility suite that identifies broken application contracts before an inference-stack rollout.
Sources
Purple and yellow clay channels contain cream beads beside a mint-green block with two round openings; one channel has a round cream tip, and a purple hand holds the square-tipped yellow connector above the other opening.
On this page5 sections

The old server returns a timeout of 30. The new server returns "30". To a reader, both say thirty. To the application, one is a number and the other is text, and the text is rejected. This hypothetical mismatch is small enough to miss in a chat and large enough to break a migration.

A r/LocalLLaMA discussion asked how people test the same open model across engines, quantizations, chat templates and hardware. Replies pointed toward a practical approach: pin the exact stack, use a small fixed suite and retain raw outputs. The discussion helps discover questions; it does not establish universal compatibility for a runtime.

Keep the request, raw reply and application result together. That record can show what the consumer needed and where it failed, instead of leaving a colleague with “the new model looks different.”

The model performs inference when it computes an answer from input. A runtime loads the model and runs that computation. A chat template arranges messages into the model’s expected format, while an API adapter packages the exchange for the application. Quantization saves resources with lower-precision numbers and can change the calculation. Any of these can affect what arrives at the client.

You need a test endpoint that accepts a saved request and a way to inspect its reply, not a training project. Begin with what your application must consume, then trace backward to the pieces producing it.

Begin with the value the application must read

Can your code read the reply? Check the fields, types, tool name and ending it expects. Elegant prose cannot rescue an integer returned as text, and a similarity score might barely notice the difference.

Each interface needs checks suited to its consumer. A tool call names a function and supplies arguments: test the name, argument types, permitted order and a case where no tool applies. Structured output must satisfy the expected field-and-type rules, or schema, and a defined rejection path. A stream arrives in pieces, so inspect event order, ending, usage fields, cancellation and partial failure. Context handling needs checks for the template, special tokens, truncation policy and tested length.

The OpenAI evaluation guidance recommends task-specific, logged, continuous evaluation rather than impressions. Here the task is not merely producing good prose. It is preserving the application contract for this workload.

Pinned model, template, runtime and hardware feed fixed tool, JSON, streaming, stop and long-context tests before review and a rollout decision.
The reader may see the same timeout while the client receives the wrong type. Saved requests, raw replies and pinned serving versions help locate that change.

Leave a reproducible route back to the reply

Hand the timeout failure to a colleague with “same model, new server,” and they still have several places to search. Record the model repository and immutable revision or file hash; tokenizer files and chat template; quantization method and parameters; engine and backend-library versions; driver and accelerator; sampling settings and seed behavior; and API adapter. Each describes a step between the original request and the received value.

Now the colleague can ask whether the quotation marks were already in the model’s output or appeared afterward. An adapter can change an outgoing reply. A template can add role or reasoning markers absent from the visible prompt. Numerical backends and quantization can change computation. An omitted option may also have a runtime default: record its resolved value. The saved pieces make the search specific.

The vLLM OpenAI-compatible server documentation and SGLang API documentation describe their compatibility surfaces. An interface label still cannot guarantee every combination of model, template, extension and client. Documentation supplies a starting contract; the suite examines the exact combination you run.

Make each test fail for a useful reason

For each saved request, state what the consumer must be able to do with the result and retain the raw reply. A tool call, a JSON object, a stream ending and a nearly full conversation need different questions. Name the model, template, runtime and hardware with each run.

Include a valid tool call, ambiguous selection, recovery from malformed arguments, strict JSON, cancellation, a stop-sequence edge, nearly full context and a repeated conversation. Sanitized incident cases belong when they expose an actual fragile point. Every case should reveal a difference that matters to the client.

Imagine a twelve-case suite for an internal incident-triage assistant. The reference produces an integer timeout; the candidate produces the same digits as a string. A person sees almost the same response. The typed dispatcher rejects one. That concrete failure tells us more about compatibility than an average similarity score.

This synthetic assertion isolates the decoded argument’s type. It calls neither runtime:

reference = {"timeout": 30}
candidate = {"timeout": "30"}

def valid_timeout(arguments):
    return type(arguments.get("timeout")) is int

assert valid_timeout(reference)
assert not valid_timeout(candidate)

Both assertions pass: the reference has the required integer type and the candidate does not. Both objects can be valid JSON; valid syntax does not enforce this application’s integer requirement. Python also treats booleans as integers for some checks, so the exact comparison intentionally excludes True. Keep the original reply with the result to determine whether generation, an adapter or another serialization step changed the value.

Use exact assertions for exact interfaces: JSON types, event order, stop state and tool names. Natural-language explanations may instead need required facts, prohibited claims or a review rubric. Requiring identical bytes from behavior that was never deterministic can create failures the application does not care about.

Keep the comparison narrow enough to explain

To isolate the runtime change, keep model revision, template, quantization, prompts, sampling and hardware steady. To examine quantization, hold the engine and template steady instead. Sometimes migration forces several changes together. That is a comparison between bundles, not proof that one component caused the difference.

For the trial, start both servers cleanly and warm them under the same policy. Send the saved request to each. Retain response bytes and available token identifiers with timing, memory, warnings and server errors. A faster answer that still quotes the timeout is unusable. A correctly typed answer that arrives too late is a different problem. Establish application use before comparing performance.

Snapshots can change model behavior too. The OpenAI API reference recommends pinned versions and evaluations where consistent behavior matters. Open models give operators more artifact control, but a floating tag is still not a reproducible baseline.

Locate the mismatch before choosing a runtime

Follow the value. A string already present in raw model output points toward generation, the template or structured-output handling. An integer there but a string at the client points toward the adapter. Expected wording variation is a different category. Classify only after the request and reply show where the change appeared.

If an action-bearing response differs and the team cannot determine whether it is harmless wording or a broken rule, resolve that before rollout. A mostly plausible transcript cannot settle an unknown timeout type or tool name.

Keep a comparison matrix with immutable component IDs and the review date. Each row describes one setup under one suite. A new model, template, backend, driver, hardware target or client creates a new setup to compare. Earlier replies remain useful when a regression appears.

The model-and-harness benchmark guide explains attribution when model and orchestration change together. The shadow-test migration guide compares candidates without production authority. Keep those comparisons beside the raw compatibility results. If thirty becomes text again, the rollout decision can name the broken contract and the exact version to restore.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next