A modern chat model is a giant table of numbers, and the size of that table decides which graphics card can run it. Quantization rewrites the table with smaller numbers so the same model fits cheaper hardware. This guide turns that idea into a shopping list: what each quantization level costs in VRAM, which one to pick for 4, 6, 8, 12 or 16 GB cards, and what you give up at each step. If you want the one-paragraph definition first, read what quantization is, and if you want the field report from a real 6 GB card, read what fits on an RTX 2060.
What quantization actually does
Training produces model weights as roomy 16-bit floating point numbers. Quantization maps blocks of those weights into compact integers of eight, six, five, four or even three bits, plus one scale factor per block so the small numbers can be stretched back close to their original values at inference time. The model keeps most of its skill because language-model weights tolerate rounding far better than intuition suggests, and the payoff is large: memory use falls roughly in proportion to the bit width, and the traffic between VRAM and the compute cores falls with it, which is why quantized models also answer faster. The llama.cpp project documents the common GGUF quantization types and is the reference implementation most local tooling builds on, while the Hugging Face quantization overview explains the same trade from the Transformers side, including the GPU-focused GPTQ and AWQ families maintained at AutoGPTQ and llm-awq.
Three vocabulary notes save confusion later. First, the trailing tags on a model name are the recipe: Q4_K_M means a 4-bit K-quant of medium size, Q5_K_M the 5-bit sibling, Q8_0 a simple 8-bit rounding, and Q3_K_S an aggressive 3-bit small variant. The K-quants mix bit depths across layer types instead of rounding everything equally, which is why a Q4_K_M reads noticeably better than the older flat Q4_0 at nearly the same size. Second, quantization is not distillation or pruning: the parameter count stays the same and only the precision changes. Third, quants labeled imatrix or i-quant were calibrated against sample text to protect important weights, so they usually beat their non-calibrated namesakes by a small but real margin. The llama.cpp wiki walks through these variants with measured perplexity comparisons for readers who want the numbers behind the ranking.
The VRAM math
Estimate any model with one multiplication: parameter count times bytes per parameter, plus room for context. A 7-billion-parameter model at 16-bit precision needs about 14 GB (seven billion times two bytes), which is why the full-precision model never fit consumer cards and the quantized one does. The same 7B model drops to roughly 7 to 8 GB at 8-bit, about 5.5 GB at 6-bit, near 5 GB at Q5_K_M, between 4 and 4.5 GB at Q4_K_M, and close to 3.5 GB at 3-bit. You can confirm the pattern on any model listing: the Ollama Qwen2.5 page and the Ollama Mistral page publish the download size of each tagged quant, and the underlying weights live on cards such as Qwen2.5-7B-Instruct and Mistral-7B-Instruct where the parameter counts are stated.
Then add the part beginners forget: the KV cache. Every token of context holds key and value vectors for every layer, so cache grows with context length, not with cleverness. A 7B model with a 4096-token window typically reserves on the order of half a gigabyte to a gigabyte for cache on top of its weights, and doubling the window roughly doubles that reservation. This is why a model that fits at 2K context can crash at 32K on the same card, and why the Ollama FAQ documents the context-length setting explicitly: context is a VRAM dial, not a free upgrade. Practical rule: budget weights plus one gigabyte of headroom for cache and the embeddings model described in our embeddings guide, and if the card still overflows, shorten context before reaching for a smaller quant.
Which quant for which VRAM
The table below assumes a chat model plus a resident embeddings model and sane headroom. Sizes are approximate because every family differs slightly; check the tagged size on the download page before pulling.
| Your VRAM | Comfortable pick | Example |
|---|---|---|
| 4 GB | 3 to 4B at Q4, or 7B at Q3 | Phi-3 Mini Q4, documented at Ollama Phi-3 with weights at Phi-3-mini-4k-instruct |
| 6 GB | 7 to 8B at Q4_K_M | Qwen2.5 7B or Mistral 7B Q4, the daily drivers in our RTX 2060 report on the 6 GB card NVIDIA specs at the RTX 2060 page |
| 8 GB | 8 to 9B at Q5_K_M, or 7B at Q8 | Gemma 2 9B Q4 with clipped context, weights at gemma-2-9b-it |
| 12 GB | 13 to 14B at Q4_K_M | A 13B Q4 fully resident with room for long context |
| 16 GB | 24B at Q4, or 32B at Q3 to Q4 | Small reasoning models with partial offload as fallback |
Two upgrades change the answer. First, Q5_K_M costs roughly fifteen percent more VRAM than Q4_K_M and buys visibly cleaner prose on long answers, so it is the right default on any card with slack. Second, Q8_0 is nearly indistinguishable from full precision for most drafting and editing, which makes a 7B Q8 the quality pick on 8 GB cards even though a larger Q4 model would also fit. When the model still does not fit, partial GPU offload spills some layers to system RAM, a technique llama.cpp supports layer by layer: usable for overnight polish at reduced speed, miserable for chat, exactly as measured in the RTX 2060 offload notes.
What you actually lose
Lower precision costs some accuracy, and the cost curve bends the same way every time: 8-bit is close to lossless, the K-family 6-bit and 5-bit cost a little, 4-bit K-quants cost a bit more but stay strong, and 3-bit is where the dulling becomes hard to ignore on demanding prompts. The damage concentrates in exactly the places you would expect: rare vocabulary, long chains of arithmetic, and staying in character across very long outputs. For blog drafting, summarising, title generation and HTML fixes, a good 7B Q4_K_M is routinely indistinguishable from its full-precision twin, which is why our whole editorial loop runs on one. For evaluation harnesses, published comparisons, or the base of a fine-tune, stay at 8-bit or full precision so the quant never becomes a confounder. Calibrated i-quants narrow the gap a touch at every level and are worth preferring whenever the download page offers them.
Run it yourself
With Ollama installed, the whole experiment is two commands and one measurement:
ollama pull qwen2.5:7b-instruct-q4_K_M
ollama run qwen2.5:7b-instruct-q4_K_M "Explain Kari's law in three sentences."Watch real usage in a second terminal with nvidia-smi --query-gpu=memory.used --format=csv -l 1 while prompting, and compare against the tagged size on the model page. To quantize your own GGUF instead of downloading one, the llama.cpp quantizer follows the documented recipe in its README: convert to full-precision GGUF first, then step down to the target type, then sanity-check perplexity before trusting the file. Keep one resident chat model at a time on small cards, cap context at 4096 tokens for 7B models, and treat anything under twenty tokens per second as a signal the model is too big for the task rather than a speed to endure.
Keep going
Quantization answers what fits; the rest of the local stack answers what it does. Next read embeddings on consumer GPUs to keep semantic search resident beside your chat model, then RAG on your own machine for the full retrieve-then-write loop. The companion notes are small models on a 6 GB card and the local agent team that turns these models into a shipping workflow, the glossary entries are quantization and embeddings, and the whole series lives under the local-AI topic.