On this page4 sections
There are two working graphics cards in the machine, and neither can hold the model you want. Infermeld explores putting some of the model’s layers on one card and the rest on the other. That can make the pair useful together, but it does not turn the memory printed on their boxes into one freely shared pool.
The experimental v0.1.0 wrapper launches a separately built llama.cpp server, exposes device and layer choices, and supplies preparation checks and a thermal watchdog. The engine and model weights come separately. A maintainer discussion appeared October 5; the release record timestamps the release at 02:32 UTC that day, still October 4 in New York. This is analysis of that release, not a new October 9 launch.
The smaller card still has to fit its share
Imagine a local log-analysis assistant using a 16 GB AMD card and a 10 GB NVIDIA card. A 3:2 split looks plausible as a first question, not a measured fit or optimum. Each card must hold its assigned work, including the space needed while the assistant reads and answers. Spare AMD memory cannot satisfy an oversized allocation on the NVIDIA device.
One server process distributes layers (parts of the model’s calculation) between the selected devices. This route reaches AMD through Vulkan and NVIDIA through CUDA. The GGUF model file carries weights and metadata. The pinned server reference describes layer and key/value-cache placement across GPUs. Those caches keep intermediate results from text the model has already processed, so the model can reuse that work as it generates an answer. A longer log packet can mean more of those results to store. Runtime buffers, the temporary space used while calculations run, need additional room too.
Now lengthen the log packet. Context and buffer reservations can change which split fits even though the cards’ advertised capacities stay the same. Infermeld exposes the choices; it does not automatically search for the best arrangement. The operator still has to leave each device enough room for its own share.

Loading the model is the first result, not the last
The project’s recorded attempts make that distinction concrete. On its RX 6900 XT and RTX 3080 pair, a dense Qwen3.8-27B Q4 artifact exited during loading at a 3:2 split with 131,072 tokens reserved. A separate 2:1 attempt loaded and passed four short checks. The capacity record keeps those outcomes distinct.
The second attempt went farther, but short answers do not fill the reserved conversation space. Can our assistant ingest the actual log packet and recover what matters? The evidence guide distinguishes allocation and short serving from filled-context ingestion, retrieval quality and sustained throughput. Reserving room for 131,072 tokens demonstrates none of those larger tasks by itself.
Another successful launcher record uses a different pinned artifact, Qwen3.6-35B-A3B Q4, with an 8,192-token reservation. Four short checks passed with speculative decoding off and with explicit MTP4. Multi-token prediction proposes draft tokens using supported model data and requires extra memory. These results concern that model and reservation, not qualification of the larger log packet or a different hardware pair.
Two backend names can point to one physical card
Before selecting a split, match device names to the actual cards. Vulkan can expose an NVIDIA card too, so Vulkan0 and CUDA0 might be two routes to the same piece of hardware. Two names are not proof that the intended AMD-and-NVIDIA pair was selected.
The build guide pins a compatible CUDA-plus-Vulkan engine revision. This route needs Linux, Python 3.11 or later, a separately compiled compatible server and a verified local model. The wrapper installs no drivers and fetches no weights. With prerequisites ready, inspect the inventory before confirming the proposed launch:
python3 infermeld.py devices --server ./llama.cpp/build/bin/llama-server
python3 infermeld.py serve \
--server ./llama.cpp/build/bin/llama-server \
--model ./models/model.gguf \
--devices Vulkan0,CUDA0 --split 3,2 --confirm-devices \
--ctx-size 8192 --ubatch 32 --dry-run
These are example paths and identifiers. In --dry-run, the launcher prints the proposed argument array without starting inference. --confirm-devices records acknowledgment; it cannot identify the physical cards for you. The preflight tool checks flags, libraries and inventory. Its GGUF magic check recognizes the expected file opening, not complete validity, so verify the model checksum independently.
Once running, the watchdog monitors AMD junction temperature. It samples every 0.1 seconds, defaults to an 85°C trip threshold and stops this invocation’s server process group when triggered. Sampling and shutdown take time: 85°C is a software trip condition, not a hard temperature ceiling. Missing or unreadable sensor data prevents startup or stops the owned server. Adequate cooling remains necessary; the watchdog does not add any.
Compare the splits with the same log packet
Use the same representative packet for each feasible split. The Linux tuning approach holds engine, model, prompts, context reservation, output cap and arrival pattern steady. Giving the faster-looking setup a shorter packet changes the job, not merely the speed of completing it.
Watch through the answer’s end. First-text and complete-response latency measure different parts of the wait; accepted diagnoses, errors and guard stops show whether the request finished usefully. The streaming latency guide separates these intervals without counting network chunks as tokens. A quick beginning followed by an unfinished or unsupported diagnosis still leaves the operator with the logs.
Compare against an arrangement that can actually run. If one card already serves the required model well, the mixed setup needs to justify its extra complexity. If one-card loading is impossible, it provides no measured speed baseline; compare other feasible arrangements. An operator who already has an adequate server launch may need little from the wrapper.
The pair earns its place when the chosen model can read that packet, finish and produce an acceptable diagnosis. Infermeld makes placement and failures easier to inspect on the way. Combined memory specifications supplied a reason to try; the finished investigation supplies the result.
Source context
Source links appear within this article. Read them alongside the author’s analysis and evaluate the guidance against your environment.
Report an error or outdated detail