Quantization: Smaller Models, Less Memory

Quantization stores a model's weights, the numbers that make up its knowledge, at lower precision. Instead of 16 bits per weight, you might use 8 or 4. The file shrinks, memory needs drop and the model can run on a smaller GPU or a laptop, usually with a small loss in quality. It is the main reason large open models run on ordinary hardware. Checked 31 October 2026.

At a glance

What it is

The Hugging Face documentation puts it this way: quantization lowers the memory requirements of loading and using a model by storing the weights in lower precision while trying to preserve as much accuracy as possible. Weights are typically stored in full-precision floating point (fp32), and half-precision formats (fp16 or bf16) are increasingly popular given the size of today's models. Some methods go much lower: the docs' compatibility table lists methods that work at 8, 4, 3, 2 and even 1 bit.

A picture helps. Imagine every weight is a number written to many decimal places. Quantizing rounds each one to the nearest of a smaller set of values, such as 16 levels for 4 bits. Rounding throws away detail, so the model's answers can drift, and clever methods choose the rounding to keep the important weights accurate.

The arithmetic

Memory for weights is roughly the parameter count times the bytes per weight.

Worked example (our arithmetic, weights only). A model with 27 billion parameters:

Real files run a little larger, because some layers stay at higher precision and there is metadata. Our model pages show the real numbers from Hugging Face. Qwen3.8-27B is listed at about 56 GB in its 16-bit repository, close to the 54 GB estimate. Gemma 4 31B shows about 63 GB in its standard repository, and a 4-bit quantisation-aware-trained build from Google at about 23 GB. Mistral Large 4 weights come to about 1,061 GB, with an NVFP4 build of about 610 GB. See Qwen models, Gemma models and Mistral models.

The estimate leaves out two other memory users: the key-value cache that grows with the context window, and the runtime's own overhead. A model that fits at a short context can run out of memory at a long one.

Formats you will meet

The Hugging Face overview adds that some methods need calibration for greater accuracy and extreme compression of 1 to 2 bits, while others work out of the box with on-the-fly quantization. Its quantization picker tool helps choose by model, hardware and goal.

GGUF and the Q-names

If you use Ollama or llama.cpp you will see names such as Q80 or Q4K_M. They describe the scheme and rough bit width. The llama.cpp quantize README shows the two-step process:

python convert_hf_to_gguf.py --outfile gemma-4-E2B-it-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M

First convert the original model to a high-quality GGUF, then quantize that file. The README warns that quantization may introduce accuracy loss, usually measured as perplexity or KL divergence, and that a suitable importance matrix (--imatrix) can reduce it. It also warns that the --allow-requantize option can severely reduce quality compared with quantizing from 16-bit or 32-bit weights. Practical advice: download a published quant from a source you trust, or make your own from the original weights, and never squeeze a quantized file again.

Quantization-aware training

Most quantization happens after training. The alternative is to train with the low precision in mind. Google's Gemma 4 repositories include a 4-bit quantisation-aware-trained build, about 23 GB against the 63 GB standard one for the 31B model. The idea is that the model learns to work around the rounding, so a quantized version can lose less quality. Check the card of any model you plan to run; if the maker supplies a quantized build, start there. See Gemma models.

What you lose and how to check

Loss is usually small at 8 bits, modest at 4 bits and larger below that, though it varies by model and method. It shows up first in hard reasoning, long outputs, tool-call formatting and rarer languages, not in everyday chat. The only reliable check is your own task:

1. Run the full-precision model or a vendor API on 20 of your real prompts, if you can.
2. Run the quantized build on the same prompts.
3. Compare blind, and note any broken tool calls or formatting errors.
4. Try the next level up if the first quant fails a task.

For coding agents, watch tool-call syntax: small quantized models more often emit malformed edits. That matters for local coding agents with Ollama. Benchmarks usually come from full-precision runs through a vendor API, so a published score may not apply to your 4-bit build. See how to read AI benchmarks.

Choosing a level

Tooling notes: Ollama and llama.cpp use GGUF. For multi-user serving, see Ollama vs vLLM. For sizing a GPU, see GPU requirements for LLMs.

Quantization is not distillation or pruning

Quantization keeps the model's shape and changes how its numbers are stored. The llama.cpp tool does have a layer-pruning option, which is a separate operation.

Our take

Quantization is why a 27-billion-parameter model fits on a normal machine. It is also a quality trade that you should measure instead of assume. Prefer maker-supplied quants, avoid requantizing and test your own prompts at the level you plan to run. Figures here come from Hugging Face and llama.cpp documentation and from our model pages, checked on the date above. We have not benchmarked any quantized build ourselves.