Quantization & Distillation

Why This Matters

Frontier models need racks of GPUs, but most real deployments need AI on a single GPU, a laptop, or a phone. Compression is how the gap closes. When you run a local model with Ollama, use on-device autocorrect, or pay a fraction of frontier prices for a "mini" model, you are using quantization, distillation, or both. Get this wrong and you pay in one of two ways: over-provisioning (renting four A100s to serve a model that would run near-losslessly on one after 8-bit quantization, a 4× ongoing cost) or under-delivering (shipping a 2-bit model whose degraded outputs quietly erode user trust). The engineer who can compute "will this model fit, and what will it cost me in quality?" from a model card makes better deployment calls than one who only knows model names.

Everyday Analogy

Quantization is saving a photo as a smaller JPEG: same picture, fewer bits per pixel, slightly less detail. It is usually imperceptible until you compress too far, at which point artifacts appear everywhere at once. Distillation is a master chef writing a simplified recipe book for home cooks: the apprentice cannot replicate everything, but captures most of what matters for everyday dishes. Note what maps where: quantization keeps the same photo (same model, same weights, coarser storage); distillation produces a new, smaller cook who learned from watching the master.

Quantization: Fewer Bits per Weight

A neural network is billions of numbers (see AI-007). Models train in 16-bit floating point (fp16/bf16), which is 2 bytes per weight. Quantization maps each weight to a lower-precision representation: 8-bit integers (1 byte), 4-bit codes (half a byte), or below. Memory shrinks proportionally:

Why does throwing away precision work at all? Trained weights are redundant: most sit in a narrow range, and the network's behavior depends on aggregate patterns, not the 7th decimal of any single weight. Modern methods sharpen this further:

• Group-wise scaling. Instead of one scale factor for a whole tensor, weights are quantized in small groups (e.g., 64–128 weights) with their own scale, so one outlier doesn't wreck its neighbors' precision.
• Outlier handling. A tiny fraction of weights and activations carry outsized magnitudes; methods like GPTQ, AWQ, and llama.cpp's K-quants allocate them extra precision. This insight is what made 4-bit practical rather than ruinous.
• Quantization-aware training (QAT) vs post-training quantization (PTQ). PTQ converts a finished model in minutes-to-hours with no gradient updates, making it the workhorse for open models. QAT simulates low precision during training so the model learns around it; this is costlier, and used when every point of quality matters (e.g., on-device models at 2–4 bits).

The workhorse today is 4-bit PTQ: a 70B model drops from multiple GPUs to a single 48 GB card, and an 8B model (16 GB at fp16) becomes a ~4.5 GB file that runs comfortably on a laptop. Formats like GGUF (llama.cpp/Ollama) made this a one-command operation.

Quantization also buys speed

LLM token generation is memory-bandwidth-bound: for each new token, every active weight must be read from GPU memory, and the arithmetic finishes long before the reading does. Rough ceiling math for an 8B model on a GPU with 1 TB/s of memory bandwidth:

• fp16: 16 GB of weights per token → at most ~62 tokens/s (1,000 ÷ 16).
• 4-bit: ~4.5 GB per token → at most ~220 tokens/s.

Real systems land below these ceilings (dequantization overhead, KV cache reads, attention), but the proportionality holds: halve the bytes, roughly double the decode speed. This is why quantization is a latency tool, not just a memory tool.

What quantization does not shrink

The KV cache, the stored attention keys/values for your context, grows with context length and is separate from the weights. A long-context session can add many gigabytes on top of the weight numbers above: for a 70B-class dense model at fp16, a 32K-token context can add on the order of 10 GB of cache, and serving many concurrent users multiplies it per request. KV-cache quantization exists too (8-bit KV is common), but when your 4-bit model mysteriously doesn't fit, the cache is usually the culprit. Activations during inference are a smaller, transient cost, and note that quantization does nothing about parameter count: a quantized MoE still needs all experts resident (see AI-064).

Distillation: A Small Model Imitates a Big One

Distillation transfers capability instead of compressing storage. A large teacher model generates supervision, and a small student model trains to match it. Two main flavors:

• Logit/soft-label distillation (Hinton et al., 2015): the student matches the teacher's full probability distribution over next tokens, not just the top answer. The distribution's "dark knowledge" (the teacher thinking dog: 0.7, wolf: 0.25, cat: 0.01) teaches similarity structure that a hard label ("dog") cannot. Requires access to teacher logits, so it's mostly used within one organization.
• Sequence-level / data distillation: the teacher simply generates high-quality outputs (answers, reasoning traces, conversations), and the student is fine-tuned on them like ordinary training data (see AI-066 on synthetic data). This works through any API, which is why licenses often address it, and is how most open-model distillation happens. DeepSeek-R1's distilled variants were made exactly this way: ~800K R1-generated reasoning traces fine-tuned into Qwen and Llama students from 1.5B to 70B, bringing reasoning behavior (see AI-063) to consumer hardware.

This is how commercial model families work: a flagship is trained at enormous cost, then smaller "mini"/"flash"/"haiku" tiers capture a large share of its everyday capability at a fraction of the serving cost. The student never exceeds the teacher's ceiling on the distilled behavior, but for a well-scoped task (support triage, SQL generation), a 4B student distilled from a frontier teacher can be indistinguishable from the teacher at 1–2 orders of magnitude lower cost.

The Wider Compression Toolbox

Quantization and distillation are the headliners, but you will meet three supporting techniques on model cards and serving docs:

• Pruning removes weights or whole structures (attention heads, layers) that contribute little. Structured pruning of an existing model followed by a short distillation "heal" is how NVIDIA produced Minitron-style compact models from larger Nemotron parents. In practice pruning is less plug-and-play than quantization for LLMs, because quality is more fragile, so it usually appears combined with distillation, not alone.
• Speculative decoding attacks latency rather than memory: a small cheap draft model proposes several tokens ahead, and the large model verifies them in one batched pass, accepting the ones it agrees with. Output is provably identical to the large model alone, but decoding can run 2–3× faster when the draft model guesses well. Notice the family resemblance: the draft model is often a distilled sibling of the big one.
• Low-rank adapters (LoRA) aren't compression of the base model, but they compress customization: instead of storing a full fine-tuned copy per customer or task (140 GB each for a 70B model at fp16), you store megabyte-scale adapter deltas and hot-swap them over one shared base. QLoRA (see the showcase below) combines this with a 4-bit base, the two techniques of this lesson meeting in one training recipe (see AI-051 on fine-tuning).

The organizing question for all of them: which resource is scarce? Memory → quantization or pruning. Serving latency → quantization or speculative decoding. Per-task capability at small size → distillation. Many custom variants → LoRA.

Measuring the Damage: How to Evaluate a Quantized or Distilled Model

Compression claims live and die on evaluation, and the standard metrics can mislead:

1. Perplexity is the headline number and the weakest signal. Quantization releases usually report perplexity deltas ("+0.05 PPL on WikiText-2"), which are cheap to compute and fine for ranking quant formats of the same model. But small perplexity gaps can hide large task-level regressions, especially in math, code, and strict instruction following.
2. Task evals on your distribution beat benchmarks. A support-bot team should measure resolution quality on real (anonymized) tickets across q8/q4/q2, not trust MMLU deltas. Degradation is uneven: casual chat survives 3-bit; multi-step arithmetic often doesn't.
3. Check behavior, not just accuracy. Aggressive quantization can subtly increase repetition, formatting failures (broken JSON), and refusal quirks. Add format-compliance checks to your eval harness; "returns valid JSON 100/100 times" is a compression eval, too.
4. For distillation, evaluate against the teacher, not the benchmark. The question is "what fraction of the teacher's quality does the student retain on my task?", e.g., student wins or ties the teacher in a side-by-side comparison on X% of prompts. That retention number, times the cost ratio, is your actual business case.

A practical acceptance bar many teams use: the compressed model must stay within a fixed margin of the original on the product's own eval suite (say, ≤2% drop), with zero regressions on a small set of must-pass cases. If q4 passes, ship q4; the memory and speed are free wins.

Worked Example: Fitting Llama-Class Models on Real Hardware

You have three deployment targets. Do the arithmetic for each: weights only, then sanity-check headroom for KV cache and runtime (~10–20%).

1. A 70B model on a single 48 GB GPU (e.g., RTX 6000 Ada / L40S). fp16: 70B × 2 B = 140 GB, needing ~3 such GPUs. INT8: 70 GB, still doesn't fit one. 4-bit: 70B × 0.5 B = 35 GB, which fits, with ~13 GB left for KV cache and overhead. Conclusion: Q4 is the only single-card option; expect a small, usually acceptable quality dip.
2. An 8B model on a 16 GB laptop (MacBook Air-class, unified memory shared with the OS). fp16: 16 GB, no (the OS needs memory too). INT8: 8 GB, workable but tight. Q4_K_M (~4.7 bits effective): ≈ 4.9 GB, comfortable, leaving room for a few thousand tokens of context and your browser. This is exactly what ollama pull llama3.1:8b gives you by default.
3. A ~3B assistant on a phone with ~8 GB RAM, of which the app may use ~2 GB. fp16: 6 GB, impossible. 4-bit: 1.5 GB, fits the app budget. Below 4 bits (with QAT to protect quality) buys back battery and thermals; this distill-then-quantize-hard pattern is what on-device assistants like Apple's ~3B foundation model use.

The pattern that falls out: each halving of precision moves a model down one hardware class, from datacenter to workstation to laptop to phone.

Decision Framework: Which Technique, When

Ask in this order:

1. Does the full model already meet quality, and is the problem only memory/cost/latency? → Quantize. It's minutes of work, no training data needed, and INT8/Q4 preserves most quality. Start at 8-bit; step to 4-bit if you need the space and validate on your own evals.
2. Do you need a model 10–100× smaller than anything that meets quality off the shelf, for a specific task? → Distill. Generate teacher outputs on your task distribution, fine-tune a small student, and evaluate against the teacher. Budget real training effort and check the teacher's license first.
3. Deploying at the edge (phone, browser, embedded)? → Both, in order: distill to the smallest viable architecture, then quantize aggressively (often with QAT). Compression composes.
4. Tempted by 2–3 bit to squeeze one more hardware class? → Test on your task, not perplexity alone. Ultra-low precision degrades unevenly; instruction following and math typically crack before casual chat does.

Two constraints that override everything: a distilled student inherits its teacher's flaws and ceiling (a weak or biased teacher yields a weak or biased student), and some providers' terms prohibit using API outputs to train competing models, so check before you distill.

Real-World Showcase

• Ollama / llama.cpp made "run a 4-bit model locally" a consumer experience. The GGUF ecosystem hosts thousands of quantized variants on Hugging Face, and a default ollama pull hands you a Q4 model without you ever choosing a bit-width.
• Apple Intelligence runs a ~3B-parameter on-device foundation model, aggressively quantized (Apple describes ~2-bit-class weight compression with quantization-aware techniques) and trained alongside larger server models, giving private, offline features within a phone's memory and battery budget.
• DeepSeek-R1's distilled family (2025), with 1.5B to 70B students fine-tuned on ~800K teacher reasoning traces, put chain-of-thought reasoning on consumer GPUs within weeks of the flagship's release, the clearest public demonstration that distillation transfers behaviors, not just knowledge.
• QLoRA (2023) flipped quantization from a serving trick into a training enabler: fine-tuning a 4-bit frozen 65B base with small LoRA adapters on a single 48 GB GPU, work that previously demanded a multi-GPU node.

Common Misconceptions

• "Quantization always destroys quality." INT8 is near-lossless on most benchmarks, and Q4 is production-grade for most tasks; degradation becomes serious mainly at 2–3 bits. The dose makes the poison.
• "Quantization and distillation are the same thing." Quantization keeps the same model in fewer bits; distillation trains a new, smaller model. One is compression of storage, the other transfer of capability.
• "A 4-bit 70B model is worse than a fp16 13B model." Usually the opposite: a larger model quantized to 4-bit typically beats a smaller model at full precision at similar memory (35 GB vs 26 GB). Parameters retain more capability than precision does.
• "The student can out-reason its teacher on the distilled task." The teacher's quality is the ceiling for what distillation alone transfers; students inherit the teacher's errors and biases too.
• "If the weights fit in memory, the model fits." The KV cache grows with context and can add gigabytes; long-context workloads fail on machines where the weight math said "fits."
• "Distilling from any API is fine, since it's just outputs." Several providers' terms restrict training competing models on their outputs; this is a licensing decision, not just a technical one.
• "Perplexity barely moved, so the quantized model is fine." Perplexity is a coarse average; a +0.05 change can coexist with broken JSON output or a real drop in multi-step math. Run task-level evals on your own workload before shipping.

Try It Yourself

1. If you can install Ollama: pull the same model at two quantization levels (e.g., llama3.1:8b-instruct-q4_0 and q8_0), ask both identical questions, and compare file size, tokens/second, and answer quality. Then try a multi-step math problem; degradation shows up there first.
2. If not: open any model's page on Hugging Face and read the GGUF quantization table. You can now interpret every row (Q4_K_M, Q8_0, Q2_K…) and predict which fits your hardware using bytes-per-parameter × parameter count.
3. Do the worked-example math for your own machine: how much RAM/VRAM do you have, and what is the largest model you can run at Q4? At Q8? Write down both numbers; that is your personal deployment envelope.

Mini Project

Run a small quantization study and a toy distillation. Part 1: pick one model (e.g., an 8B instruct model) at three precisions (q8_0, q4_K_M, q2_K via Ollama or llama.cpp). Build a 15-prompt eval: 5 factual, 5 multi-step math, 5 instruction-following (e.g., "reply in exactly 3 bullet points"). Score each precision on accuracy and measure tokens/second; plot quality vs speed and identify your sweet spot. Part 2: use the largest model you can run as a teacher to generate 100 Q&A pairs in a narrow domain, fine-tune nothing; instead, put 10 of those pairs in a small model's prompt as few-shot examples and measure how much the small model improves on held-out questions. You will have demonstrated the quantization quality curve and the essence of capability transfer, all on one machine.

Key Takeaways

• Quantization = same model, fewer bits: 70B × 2 bytes = 140 GB at fp16 vs ~35 GB at 4-bit; each halving of precision moves a model down one hardware class.
• Because LLM decoding is memory-bandwidth-bound, quantization buys speed as well as memory, roughly proportional to the byte reduction.
• 8-bit is near-lossless, 4-bit is the practical sweet spot, 2–3 bit needs task-specific validation (and often QAT); always evaluate on your own workload.
• Distillation = a new small student trained on a large teacher's outputs or distributions; it transfers behaviors (even reasoning, per DeepSeek-R1's distills) but never exceeds the teacher's ceiling, and API licenses may restrict it.
• The two compose: distill to a small architecture, then quantize it. This is the standard recipe behind phone assistants and every cheap "mini" model tier.
• Compression choices are deployment choices: budget weights and KV cache against memory, bandwidth against latency, and quality against your own evals.