What Is Quantization?

Store weights with fewer bits. Large memory and speed savings for small quality loss, and the main reason big models run on small hardware.

A parameter is a number, and numbers need bits. Training typically uses 16 bits per parameter, so a 70B model needs about 140GB just to hold its weights — more than any single consumer GPU.

Quantization stores them with fewer bits. At 4 bits, that same model needs roughly 35GB and fits on hardware you can actually buy.

The tradeoff

Fewer bits means fewer distinct representable values. 16 bits gives tens of thousands of gradations; 4 bits gives sixteen. Every weight must round to the nearest available value, and that rounding is error.

What makes it work is that neural networks tolerate this remarkably well. The information is distributed across billions of parameters, so small independent errors largely wash out rather than compounding.

The empirical shape: 8-bit is nearly lossless, 4-bit costs a little quality, and below 4-bit degrades noticeably. The exact point varies by model and method.

Why it speeds things up

The obvious benefit is memory. The less obvious one is speed, and it is often larger.

Generating a token requires reading every active parameter from memory. On modern hardware, that memory traffic — not the arithmetic — is the bottleneck. Halve the bytes per weight and you roughly halve the data moved per token.

So quantization makes generation faster even when memory was not scarce. This is the same insight behind FlashAttention: memory movement is what limits real throughput.

How it is done

Scaling per group. Weights are not uniformly distributed, so a single scale factor across a whole matrix wastes precision. Instead, split weights into small groups and give each its own scale. More scale factors to store, much better accuracy — this is standard.

Outlier handling. A small number of weights and activations are far larger than the rest, and they matter disproportionately. Naively quantizing them destroys quality. Methods either keep outliers in higher precision, or transform the weights to redistribute magnitude before quantizing.

Calibration. Run sample data through the model, observe the actual value ranges, and choose quantization parameters to fit. A few hundred representative examples is usually enough, and using data resembling your real workload helps.

Post-training versus during training

Post-training quantization takes a finished model and compresses it. Fast — minutes to hours — and requires no training infrastructure. This is what most people use, and named methods differ mainly in how they handle outliers and choose scales.

Quantization-aware training simulates the rounding during training so the model adapts to it. Better results at very low bit widths, at the cost of a training run.

For 8-bit and 4-bit, post-training is generally sufficient.

What gets quantized

Weights are the main target — static, so they can be compressed once and reused.

Activations are harder. They change per input, have wider dynamic range, and contain more outliers. Many deployments quantize weights only.

KV cache quantization matters for long context, where the cache can exceed the model’s own memory. Effective, and a distinct decision from weight quantization.

Practical notes

Quality loss is uneven across tasks. Aggregate benchmarks may barely move while a specific capability degrades. Multi-step reasoning and code generation tend to suffer first, since they have less error tolerance per step. Test on your task with an eval, not on published averages.

A larger quantized model usually beats a smaller full-precision one at equal memory. A 4-bit 70B model generally outperforms a 16-bit 13B model. This is the practical decision quantization actually enables.

QLoRA combines cleanly with it — quantize the frozen base, train adapters in higher precision. Quantization error in weights that are only read matters far less.

Hosted APIs may already quantize. Providers do not always disclose serving precision, which is one reason model behaviour can shift without a version change.

What to remember

• Quantization stores parameters in fewer bits — 8-bit near-lossless, 4-bit slightly lossy, below that noticeably degraded.
• It works because error is distributed across billions of parameters rather than compounding.
• Speeds up generation by cutting memory traffic, which is the real bottleneck.
• Per-group scales and outlier handling are what make low bit widths viable.
• A larger quantized model usually beats a smaller full-precision one at the same memory budget.
• Test your own task; aggregate benchmarks hide uneven degradation.

Next: Model Distillation