Model Quantization
A 70B parameter model in full 32-bit floating point requires ~280GB of VRAM - far beyond any consumer or workstation GPU. Quantization compresses model weights to lower numerical precision, dramatically reducing memory requirements and inference latency, at a small and often acceptable quality cost.
How Quantization Works
During training, model weights are stored as 32-bit or 16-bit floating-point numbers. Quantization maps these to lower-precision integer or floating-point types. The key insight is that most model weights cluster tightly and can be represented accurately with fewer bits.
| Precision | Bits per weight | 7B model size | Quality vs FP16 |
|---|---|---|---|
| FP32 | 32 | ~28 GB | Baseline (rarely used for inference) |
| FP16 / BF16 | 16 | ~14 GB | Near-identical; standard for GPU inference |
| INT8 | 8 | ~7 GB | ~1–2% quality loss; undetectable in most tasks |
| INT4 (Q4) | 4 | ~3.5 GB | ~3–5% quality loss; fits on consumer GPUs |
| 2-bit (Q2) | 2 | ~1.8 GB | Significant degradation; only for extreme memory constraints |
Quantization Formats
GGUF (llama.cpp)
The standard format for running quantized models on CPU + GPU mixed inference. Used by Ollama, LM Studio, and any llama.cpp-based tool. Files named likemodel-Q4_K_M.gguf - the suffix encodes precision (Q4 = 4-bit) and variant (K_M = k-quants medium, balancing quality and size).
Best for: local deployment on Mac (Metal) or CPU+GPU mixed setups
GPTQ
GPU-only quantization. Post-training quantization that uses a calibration dataset to minimize quantization error. Higher quality than naive INT4 for the same bit width. Popular for vLLM deployments.
Best for: GPU servers where you want high throughput at 4-bit precision
AWQ (Activation-aware Weight Quantization)
Identifies the most important weights (those that activate frequently) and preserves their precision while quantizing the rest more aggressively. Better quality than GPTQ at the same bit width. Supported by vLLM, TGI, and TensorRT-LLM.
Best for: production GPU inference where quality matters more than quantization speed
Which Quant to Use?
- Local Mac / CPU: Q4_K_M GGUF - best quality/size balance; runs well on Metal
- Local GPU (8–12GB VRAM): Q4_K_M GGUF or AWQ INT4 for CUDA
- Production GPU server: AWQ or GPTQ via vLLM - higher throughput than GGUF
- Cloud API inference: Provider handles quantization internally; use full-precision tier for quality-critical tasks
When Quality Degradation Matters
For most text generation tasks, INT8 and even Q4 quantization produce outputs indistinguishable from FP16. Quality drops become noticeable for:
- Complex multi-step reasoning (where small errors compound)
- Code generation for large programs with many dependencies
- Tasks requiring precise numerical output
The practical advice: start with Q4_K_M for local deployment. Benchmark your specific task before assuming you need a larger quantization or full precision.