What Basalt changes in a Blackwell local inference stack

Four engineers want to submit long code investigations to one local server. In the hypothetical case below, each request needs 64K tokens of context and must finish within the team’s latency objective. Whether the server can complete all four tells them more than its fastest isolated token count.

Basalt targets that kind of local inference with a deliberately narrow design: Qwen3.8-Flash-Next on NVIDIA Blackwell GPUs. It can keep model parts that do not fit on the main GPU in system RAM; its Linux path can use a second GPU too. Linux is the tested benchmark path, while Windows support has narrower limits.

The launch thread in r/LocalLLaMA reported 354 prose tokens per second, 665 structured tokens per second, and 7,317 prompt tokens per second at 64K context for an IQ3_XXS pack on an RTX 5090 plus RTX 5060 Ti. Those are the maintainer's measurements on a stated machine, not an independent benchmark. The useful operational question is what the design changes about fit, concurrency, and the measurements needed before using it behind a real service.

A narrow engine trades portability for control

Before the four engineers compare speeds, they have a simpler question: can their machine run this engine at all? Basalt is not offering to move the same service freely among GPU families. A fleet of similar Blackwell workstations can benefit from code shaped around their hardware, memory transfers and file format. A different GPU family would require a different serving plan, not just a slower setting.

The Basalt repository couples a C++ and CUDA engine to a Python server offering OpenAI-compatible and Anthropic-compatible endpoints. Conversion tools, tests and a single-file model format support that serving route. For the engineers, the immediate constraint is the machine underneath it: the documented Linux path needs x86-64 with AVX2, a Blackwell GPU with compute capability 12.0 and CUDA 13. System RAM must also have room for model parts, called experts, that cannot fit in GPU memory, or VRAM. A matching card alone does not settle that last question.

Windows is a separate choice with narrower evidence. The current platform guide supports Windows x64 on a single Blackwell GPU, lists memory and process limitations, and says Windows quality measurements and headline benchmarks are not yet available. AMD, Intel, older NVIDIA generations and macOS remain outside the documented target. The optimization that helps this workstation does not supply a fallback platform.

There is a second fit to establish before the engineers send their investigations: the model file must suit the engine. Basalt repacks supported GGUF weights into a .basalt file. The repository says it copies tensors rather than requantizing them, apart from an explicitly recorded compatibility conversion. A new container for the weights is not, by itself, a new lower-precision model. Keep the pack's identity with the run so that distinction survives the comparison.

The pack also carries the material the engine needs around those weights. Those details matter when preparing the server, but the team's next performance question remains simpler: where will the model's work live once four people are using it?

Pack contents and licenses for the server inventory

The single-file pack contains tokenizer data, the chat template, MTP drafter, expert profile, vision projector, provenance and checksums. The engine is MIT-licensed; model packs retain the Qwen Community License. Repacking does not merge those rights, so record the engine and model pack as separate artifacts.

Experts can move without becoming free

Qwen3.8-Flash-Next is a mixture-of-experts model: it has specialized parameter groups, called experts, and selects a subset as it processes tokens. That reduces the work needed for each token, but the unselected experts still need somewhere to live. Basalt keeps the dense work and frequently used data close to the GPU, while experts outside its memory budget can live in system RAM and be streamed as needed. On Linux, a second Blackwell card can provide another expert tier. This makes a large model fit on more workstation configurations, with transfer time becoming part of the tradeoff.

Watch what happens as the engineers’ conversations grow. Attention state needs more memory, leaving less room for experts on the GPU. The next token may have to wait for more transfers from slower memory. Additional sessions add competitors for that same space. A successful startup has not yet answered the four-user question.

The server makes that competition visible. Its configuration can set the model, GPUs, context, KV format, expert-cache behavior, speculative decoding, and VRAM reserve. The key-value, or KV, cache holds intermediate attention data for an active request; as its conversation grows, that state needs memory alongside the experts. Startup output reports where KV cache is allocated and warns when batch slots leave too little room for the expert set. Those warnings are evidence about the selected configuration, not proof that another split is faster.

Concurrency divides context unless you pay for more cache

Basalt's --parallel option creates up to eight batch slots. By default, the configured maximum context is shared among them. Four slots under a 262,144-token maximum therefore receive 65,536 tokens each. Requests exceeding a slot's limit are refused with a 400 response. The alternative --slot-full-context gives every slot the full maximum, but multiplies KV-cache demand and reduces room available to the expert cache.

Four slots only help these engineers if each still holds the investigation they must send. Reducing the permitted context to raise throughput would change the job being compared. Keep the 64K requirement and watch waiting, refusals and completed investigations.

Prompt processing and decoding also answer different questions. The maintainer reports high prefill rates and separate prose, structured, and chat decode results. A workload dominated by fresh 64K prompts may care more about time to first useful text. A queue of short structured decisions may care about completed decisions per second and tail latency. Copying the largest number into a capacity plan loses that distinction.

Reproduce the claim with your accepted workload

Send the four engineers' investigations on a fixed arrival schedule, with representative prompt lengths and output limits. Keep their arrivals and request lengths steady while comparing slot and expert-cache settings. Which requests finish within the objective, which wait, and which are refused? A refusal belongs in that account, not outside the benchmark because it never produced text. The Linux performance guide develops that comparison.

To make the result repeatable, keep a versioned configuration and one identified model pack with the run. Another engineer needs to recover the exact engine commit and pack checksum, then the GPU models, driver and CUDA versions, power limit and RAM. The context policy, slot count, speculative settings and prompt set explain what those resources were asked to do. Otherwise a changed pack or power setting can quietly become the apparent benefit of a new slot count.

Use the project's tests before the service benchmark. Some kernel parity tests require GPU data or locked-memory configuration, while file-format and cache tests do not. Keep those prerequisites with the test result rather than treating an unrun check as a pass.

Follow each investigation from arrival to its outcome. Time to first text tells the engineer when the reply begins; completed-response latency tells them when the answer is available. Accepted completions, refusals, cancellations and queue depth show what happened to the rest of the offered work. Keep warm and cold results separate. A repeated prompt may reuse a prefix and become inexpensive, but that is a cached workload, not fresh operational evidence. The streaming LLM latency guide keeps first text, normal completion, refusal and interruption as separate outcomes.

Then read the investigations themselves. A fast answer that mishandles a log or proposes a bad code change has not served these engineers. The project's model-card measurements help compare packs, but answer narrower questions. Kullback-Leibler divergence and top-1 comparisons describe closeness to a Q8_0 reference. Each pack's perplexity instead measures its own fit to the stated text; it is not a distance from Q8_0. Those numbers cannot establish that a quant is accurate enough for log diagnosis, code changes or structured incident updates.

A held-out task set supplies that application check: investigations the configuration was not adjusted to fit, with failures kept beside the exact pack that produced them. Now the team can compare both parts of its original request: did all four answers arrive in time, and were they useful answers?

The server boundary still belongs to the operator

Basalt listens on 127.0.0.1 by default. Its documentation warns against exposing the server beyond that address without an API key, and the API key cannot be placed in the configuration file. A compatible endpoint is convenient, but compatibility does not supply authentication, tenant isolation, admission control, or production observability by itself.

If the required model already runs well on a broader engine, Basalt's narrower support needs a workload-specific benefit to justify another serving path. If the model does not fit or concurrency is the limiting factor, the specialized memory and batch controls may make a useful experiment possible.

Return to the engineers waiting for their investigations. If Basalt lets all four finish within the required time where the broader engine could not, the specialization bought something concrete. If both already do that job, another serving path needs another benefit worth its hardware and release constraints.