Opens in a new tab
Explainer

Four long investigations, one Blackwell workstation: trying Basalt

In brief

Basalt makes specific choices about where a large model lives. Follow four hypothetical users to see how those choices affect memory, waiting and useful answers.

8 min read

Is this the right starting point?
Before you read
Experience operating GPU inference services and measuring workload capacity.
What you will take away
Evaluate Basalt’s hardware specialization using context fit, concurrent accepted work and versioned quality and latency checks.
Sources
Copper-colored blocks sit in water-filled channels cut through dark rock, between a large ribbed metal structure on the left, a smaller one at the back, and a broad basin on the right.
On this page5 sections

Four engineers have code investigations ready to send to one local server. Each needs room for 64K tokens, the pieces of text the model handles, across its context: the material and conversation it can work with at once. Each answer also has to arrive within the team’s allowed response time, its latency objective. In this hypothetical afternoon, nobody is asking how quickly the server can produce a clever sentence in isolation. They want to know whether all four investigations will finish in time.

Basalt takes a deliberately specialized route to that job: Qwen3.8-Flash-Next on NVIDIA Blackwell GPUs. Parts of the model that do not fit on the main GPU can live in system RAM or an optional second GPU. That gives the workstation more places to keep a large model, but moving data between them takes time. Linux remains the documented benchmark path; Windows has its own limitations.

The maintainer’s launch thread in r/LocalLLaMA reported 354 prose tokens per second, 665 structured tokens per second and 7,317 prompt tokens per second at 64K context. The setup used an IQ3_XXS pack with an RTX 5090 and RTX 5060 Ti. Those are the maintainer’s measurements, not an independent AIOpsSRE benchmark. For our four engineers, the interesting part is how the design might make their work fit, and what happens when everyone arrives together.

Start with the workstation on the desk

Before discussing speed, check whether this is a machine Basalt is meant to use. The engine is not a portable way to move the same service among unrelated GPU families. Similar Blackwell workstations may benefit from code tailored to their hardware, transfers and file format. A different GPU family calls for another serving plan, not simply a slower Basalt setting.

The Basalt repository joins a C++ and CUDA engine to a Python server, with OpenAI-compatible and Anthropic-compatible endpoints. It also supplies conversion tools, tests and its own single-file model format. The documented Linux requirements include x86-64 with AVX2, a Blackwell GPU with compute capability 12.0, and CUDA 13. There is a less visible requirement beside the graphics card: enough system RAM for the model’s expert groups that do not fit in the card’s memory, called VRAM.

Windows support has changed since this article’s October 10 source review. The platform guide, checked again November 2, now lists an optional second Blackwell GPU on Windows x64 as well as Linux, rather than the earlier single-GPU limit. It still describes Windows memory and process restrictions and says Windows quality measurements and headline benchmarks are unavailable. AMD, Intel, older NVIDIA generations and macOS are outside the documented target. A supported platform is not the same thing as a measured result on it.

Next comes the file the server will load. Basalt can repack supported GGUF weights into a .basalt file. For that GGUF route, its documentation describes copying tensors rather than changing their numerical precision, apart from an explicitly recorded compatibility conversion. Repacking is therefore not automatically a new, more compressed model. Keep the pack’s identity with the run so a file-format change does not get mistaken for a change in model precision.

The file brings more than weights. It must carry the supporting material the engine expects, and the version of the pack must agree with the engine reading it. That preparation gets us to a more immediate question: how much room will remain once four conversations are open?

What travels in the pack, and what has separate rights

The pack includes tokenizer data, the chat template, a multi-token prediction (MTP) drafter that proposes upcoming tokens, a vision projector, provenance and checksums. The current model-file documentation, checked November 2, explicitly places the expert profile with the engine rather than inside the pack; this differs from the earlier description. The engine is MIT-licensed, while model packs retain the Qwen Community License. Keep those two artifacts and their rights separate when recording the server setup.

Dense work, expert cache, KV cache and prompt buffers share the main Blackwell GPU while nonresident experts remain in system RAM and an optional peer GPU.
A conceptual view of memory placement, not a measured benchmark. Longer conversations and additional slots need attention-cache space that could otherwise hold experts.

The model can fit and still spend time moving

As the model processes a token, it selects some specialized groups of learned numerical settings, or parameters, rather than using every group. These groups are the experts in a mixture-of-experts model. Selecting fewer reduces the computation per token, but the other experts still need storage. Basalt keeps the dense work and frequently used data near the GPU. Experts outside that memory budget can stay in system RAM and move across when needed; a second Blackwell card can provide another tier. The larger model can fit, at the cost of transfers that an all-resident setup would avoid.

Now let the four conversations grow. Each needs more remembered attention data, leaving less GPU space for experts. The next token may have to wait while an expert arrives from slower memory. A second conversation competes for that space, then a third and fourth. The model loading successfully was only the beginning of the capacity question.

That remembered attention data is the key-value, or KV, cache. It holds intermediate information an active request needs, alongside the experts sharing memory with it. Basalt exposes settings for the model, GPUs, context, KV format, expert cache, speculative decoding and spare VRAM. Startup output reports cache placement and warns when batch slots leave too little room for experts. Read those warnings as information about this configuration, not as a promise that a different split will be faster.

Four slots need four usable conversations

The --parallel option creates up to eight batch slots. By default, they share the configured maximum context. With four slots and a maximum of 262,144 tokens, each gets 65,536 tokens. A request beyond its slot’s limit receives a 400 response. Alternatively, --slot-full-context gives each slot the full maximum. That multiplies KV-cache demand and takes room away from the expert cache.

Four slots do not help if they make each investigation too small to submit. Keep the engineers’ 64K requirement in the comparison. Lowering permitted context until the throughput looks attractive would answer a different question. Count the requests that wait, the ones refused and the investigations completed.

The stage of the work matters, too. Reading the prompt is called prefill; generating the reply is decoding. The maintainer reports separate prefill, prose, structured-output and chat results. Four fresh 64K prompts may leave their users waiting most noticeably for the first useful text. Short structured decisions may instead make completed decisions per second and the slowest responses more important. The largest advertised number cannot stand in for both jobs.

Measure the afternoon the engineers actually need

Send representative investigations on a fixed arrival schedule, preserving prompt lengths and output limits while comparing slot and expert-cache settings. Note which finish within the objective, which wait and which are refused. A rejected request still came from someone who needed an answer; it should not disappear from the account because no text was generated. The Linux performance guide develops this kind of comparison.

Leave enough detail for another engineer to repeat it: engine commit, model-pack checksum, GPU models, driver, CUDA version, power limit and RAM. Save the configuration with its context policy, slots, speculative settings and prompt set. Without those details, a different pack or power setting could take credit for an improvement supposedly caused by a new slot count.

Run the project’s applicable tests before the service benchmark. Some kernel comparisons need GPU data or a locked-memory configuration; file-format and cache tests have different requirements. Keep the prerequisites and actual results together. A check you could not run remains unrun, not passed.

For each investigation, record the wait until text starts and the time until the complete response is available. Accepted completions, refusals, cancellations and queue depth describe what happened to the rest of the offered work. Separate cold and warm runs. A repeated prompt can reuse a stored prefix and become much cheaper to process, but that describes a cached workload, not a fresh investigation. The streaming LLM latency guide separates first text, normal completion, refusal and interruption.

Then read the answers. A quick response that mishandles a log or suggests a bad code change has not helped the team. The project’s model-card measurements compare packs in narrower ways. Kullback-Leibler divergence, a measure of how two predicted probability distributions differ, and top-1 agreement, how often their first-choice token matches, compare against Q8_0. Perplexity instead measures each pack’s fit to the stated text, not its distance from Q8_0. None of those measurements alone establishes that a lower-precision pack is good enough for diagnosis, code changes or incident updates.

Use a held-out task set for that application check: investigations not used to tune the configuration, with failures kept beside the pack that produced them. We can now return to both halves of the engineers’ request. Did their answers arrive in time? And were those answers worth waiting for?

Decide who can reach the server

Basalt binds to 127.0.0.1 by default, the local machine’s loopback address. Its documentation warns against exposure beyond that address without an API key, and does not allow the key in the configuration file. Familiar API shapes make clients easier to connect. They do not by themselves supply authentication, separation between tenants, admission controls or production monitoring.

If the required model already does this job well on a broader engine, another specialized serving path needs a concrete benefit to justify its support burden. If model fit or concurrent requests are the obstacle, Basalt’s memory and batch controls may open a useful experiment. The machine and workload decide which situation we are in.

Picture the four engineers getting their investigations back. If Basalt lets all four finish within the required time when the broader engine could not, the specialization bought something they can use. If both already do that, speed on an unrelated benchmark is not yet a reason to maintain another hardware-specific path. Their afternoon gives the comparison its purpose.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next