Model quantization: how fewer bits make AI fit

An eight-billion-parameter model needs about 16 GB just to store its weights at 16 bits each. Represent those same eight billion values with four bits apiece, and the raw arithmetic drops to 4 GB. That is a substantial change before the model has answered a single question.

Quantization creates that saving by representing numbers with fewer possible values. It can make a useful model fit on a laptop, leave more room on a graphics processor (GPU) for requests, or reduce the amount of data moved during inference. The trade is numerical precision, and the result depends on how the conversion and the runtime handle it. Hugging Face’s quantization overview describes the underlying precision tradeoff.

The interesting question is what that smaller representation buys for the work you want to do. A model that loads comfortably but struggles with the documents it must summarize has solved only part of the problem. To choose a variant sensibly, it helps to follow one number from the original model into its smaller form, then account for everything the running application still needs.

What gets smaller?

A model’s weights are the learned numerical values used in its calculations. Training changes them; inference uses them to produce an answer. If that distinction is unfamiliar, the model-weights guide explains the training side, while the inference guide follows a request through the model.

In weight quantization, the model keeps its network structure and many or all of those values receive a more compact representation. An eight-billion-parameter model generally still has eight billion parameters. It has not become a two-billion-parameter model simply because the stored weights take one quarter of the space.

Four bits provide 16 possible bit patterns. A quantization scheme specifies how those patterns represent numbers. Some schemes use regularly spaced integer codes with a scale; others use a different set of representable values. The format therefore matters as well as the bit count. QLoRA’s NF4, for example, is a four-bit representation designed around normally distributed weights rather than a general promise that all four-bit formats behave alike. QLoRA paper

The numbers produced between layers, called activations, can also be quantized. A description such as weight-only quantization is useful because it tells you which part changed. It does not imply that every temporary value, calculation and cache in the application uses the same precision.

Follow one weight through the conversion

Consider a deliberately simple teaching example. A scheme stores a small integer code and uses a scale of 0.1, with no offset. A weight of 0.26 is divided by 0.1 and rounded: 2.6 becomes the code 3. Multiplying that code by the scale reconstructs 0.3.

The compact representation retains an approximation, not the original 0.26. Here the difference is 0.04. If a calculation multiplies the weight by an input of 1.5, its contribution changes from 0.39 to 0.45. Real layers combine many such contributions, so this little calculation explains where a change can enter without predicting the final answer’s quality.

Read diagram description

Divide 0.26 by the scale 0.1 to get 2.6, then round to the integer code 3. Multiply the code by 0.1 to reconstruct 0.30. The difference between 0.30 and 0.26 is plus 0.04.

This example uses one scale to make the mechanism visible. Real schemes can store separate scales for different groups of values, giving each group a better-fitting numerical range. They may also use an offset, often called a zero-point. Those extra values consume space, but can preserve detail that one scale across a much broader range would lose. The integer-inference quantization paper describes the mapping, and NVIDIA’s quantization guide explains how scales can be shared across groups.

The choice of range matters. If a few very large values stretch a shared range, smaller values can become difficult to distinguish. More sophisticated methods spend their limited precision carefully. AWQ uses activation statistics to identify and protect important weight channels through scaling. GPTQ uses information about a layer’s behavior to reduce the damage caused by quantizing its weights. These are different ways to improve an approximation, not evidence that every converted model will preserve every task equally. AWQ, GPTQ

Where the rest of the memory goes

For the hypothetical eight-billion-parameter model, the simplest storage estimate is:

8,000,000,000 parameters × 4 bits ÷ 8 bits per byte = 4,000,000,000 bytes

That is 4 GB in decimal units, or about 3.73 GiB. It describes only the idealized packed weight values. It leaves out scales, other quantization information, any weights retained at higher precision and file metadata. One concrete example is GGUF’s Q4_K representation: its block structure uses 4.5 bits per weight after the block information is included. A label containing “4” is not an exact whole-file-size formula. Hugging Face’s GGUF format guide

The running process needs more space again. In a transformer generating text, a key-value cache retains attention information from earlier tokens so the model can reuse it rather than calculate it all again. Longer sequences and more simultaneous requests can enlarge that working set, depending on the model and cache design. Quantizing the weights does not automatically quantize this cache. Cache quantization is a separate option with its own latency and compatibility tradeoffs. Transformers cache documentation

Suppose, in a hypothetical document-summary service, a quantized model loads successfully and answers a short greeting. The intended workload consists of long support cases, with several arriving together. The greeting establishes that the model can start and generate text. It says little about the memory needed to hold those longer requests at the same time. Capacity planning becomes much more useful when the test resembles that work.

On a machine with shared memory, the operating system and other applications also need room. On a discrete GPU, system RAM and GPU memory are separate resources. A 4 GB file alone cannot settle whether either arrangement will handle the planned workload.

Smaller can be faster, but the runtime does the work

Inference involves both calculation and movement of data. When a device spends much of its time bringing weights from memory to its compute units, a compact representation can reduce that traffic. This is one reason quantization can help even when the device does not perform every operation directly on four-bit numbers.

The runtime still needs a suitable implementation. A kernel, the low-level routine that performs an operation, may unpack the stored values and use a different precision for arithmetic. In bitsandbytes, for example, four-bit storage and the computation data type are separate settings; its documentation shows a four-bit configuration using bfloat16, a 16-bit format, for computation. Bitsandbytes integration documentation

That distinction explains why “four times smaller” does not imply “four times faster.” Unpacking takes work. Hardware support differs. A batch of requests can put different demands on the device than a single conversation. Hugging Face explicitly notes that an inference speedup is not guaranteed when selecting a quantization method. Method-selection guide

Fitting the entire workload into a more useful memory arrangement can be the main benefit even when raw token generation barely changes. Conversely, a serving setup that already fits comfortably may gain little from a more aggressive conversion. The useful comparison includes time to the first useful response, total completion time and the number of simultaneous requests that finish acceptably, alongside memory consumption.

Conversion, training and fine-tuning do different jobs

Post-training quantization starts with an already trained model and produces a lower-precision representation. Some methods use calibration inputs to observe how the model behaves before choosing the conversion. This is different from teaching the model a new subject through ordinary training. GPTQ is one established post-training approach. GPTQ paper

Quantization-aware training accounts for quantization effects during training so the model can adapt to the approximation. The integer-inference paper pairs its quantization scheme with a training procedure intended to preserve accuracy. Jacob and colleagues

QLoRA combines another pair of ideas: it keeps a quantized base model frozen while training small additional LoRA parameters. Its purpose is memory-efficient fine-tuning. Downloading a four-bit model, by itself, does not mean someone has fine-tuned it for your support categories or operating environment. The LoRA guide covers that separate adoption decision. QLoRA paper

Choose a variant for the work that must fit

Start with a model that can do the task and a format supported by the runtime and hardware you actually intend to use. GGUF is a file format, AWQ is a quantization method, and bitsandbytes is a software library that provides quantized operations. Matching the bit count is not enough to make their files or configurations interchangeable. The ninfer-ext article follows one concrete case where the model encoding and its loader have to agree.

For the hypothetical summary service, a useful comparison keeps the same source cases and asks each candidate to retain the customer’s problem, actions already attempted and unresolved questions. Include long cases and examples where one small qualification changes the meaning. Then compare the memory and completion behavior at the concurrency the service is expected to handle. This is a proposed evaluation, not a test performed for this article.

Comparing different precisions of the same base model helps isolate the conversion’s effects. Comparing a larger quantized model with a smaller higher-precision model answers a different, equally legitimate question: which complete option serves this workload better? Keep those conclusions separate so an improvement from changing the base model is not attributed entirely to quantization.

The four-bit label now tells you something precise: the representation has been made more compact. The rest of the choice comes from following that representation into the running service. The right saving is the one that leaves enough room for the actual requests while preserving the answers people need.