Quantization explained: 4-bit, 8-bit and what you lose
Last updated
Quantization stores a model's weights with fewer bits. At 4-bit a model needs roughly 0.5-0.6 GB per billion parameters (approximate) and keeps most of its quality. At 8-bit it needs about twice the memory and stays closer to the original. Small models lose the most.
Olares One → Local LLM hardware → VRAM · quantization · tokens per second → The right machine for your model size
Quantization explained: what does it do to a model?
Quantization stores a language model's weights with fewer bits each. Weights are grouped into small blocks, and each block keeps a scale factor, so the rounded values stay close to the originals. Fewer bits mean a smaller file, less memory and faster generation. The price is a small rounding error in every weight.
Speed improves because each generated token reads every active weight from memory once. Fewer bytes per weight means fewer bytes to read.
4-bit, 8-bit and what you lose: how big is the gap?
At 8-bit, a model stays close to its original quality for most tasks. At 4-bit, the common default for local use, the loss is small for chat and writing on mid-size and larger models. Below 4 bits, errors grow quickly. Small models lose more than large ones, because they have less redundancy to absorb rounding.
The loss is uneven. Arithmetic, code and long chains of reasoning show errors sooner than casual chat, and less common languages can degrade before English does. We'd run a larger model at 4-bit before a smaller one at 8-bit. The exception is code or math on a small model, where the 8-bit version is worth a test on your own tasks.
Which quantization formats will you meet?
You'll meet four families of quantized files. GGUF serves llama.cpp, Ollama and LM Studio. AWQ and GPTQ are GPU formats that vLLM loads. FP4 is a 4-bit floating-point format that newer NVIDIA GPUs run in hardware, and MLX covers Apple silicon. Each family has its own tools, so pick the format your engine reads.
GGUF names such as Q4_K_M or Q8_0 show the bit width and method. FP4 also works for image models. In Olares' own test on Olares One, FLUX.1 dev with Nunchaku FP4 made a first image in 15.51 s and later images in 8.32 s each. The KV cache can be quantized as well, which frees memory for longer context at a small quality cost.
What should you run on a 24 GB GPU?
On a 24 GB GPU such as Olares One, a 32B model at 4-bit is the largest dense model that fits with some room for context. Models of 14B or 8B fit at 8-bit with room to spare. A mixture-of-experts model like Qwen3-30B-A3B at 4-bit takes about as much memory as a 32B dense model but generates faster.
In Olares' own test, Qwen3-30B-A3B ran at 157 tokens per second with a single request on vLLM. For memory sizes see how much VRAM you need, and for speed see tokens per second explained.
Questions & answers
Is 4-bit quantization good enough?
For most chat, writing and summaries on models of 14B and up, yes. For code and math, test the 4-bit and 8-bit versions on your own tasks before you settle.
Does quantization make a model faster?
Usually. Generation reads every active weight per token, so fewer bytes per weight means more tokens per second on the same hardware.
Sources for this entry
- Olares blog: local AI hardware benchmarking (Oct 2025) (Company-published)
- Olares One FAQ (Olares docs) (Company-published)
Continue this reading path
How the related websites connect
Cite this entry
Best Local LLM Hardware. “Quantization explained: 4-bit, 8-bit and what you lose.” Content reviewed 13 November 2026.
https://bestlocalllmhardware.com/quantization-explained/