Synthetic Data & Training Pipelines

Why This Matters

"We trained a model" hides a factory. Understanding the pipeline explains why models behave as they do: capabilities come from pre-training, helpfulness from fine-tuning, judgment from preference tuning. It changes how you debug in practice: a model that doesn't know your product (a pre-training gap) needs retrieval or fine-tuning with new information, while a model that knows but answers in the wrong format (an SFT-layer behavior) usually needs only prompting or a few hundred examples; teams that can't tell these apart burn months fine-tuning for problems a system prompt would fix. It also explains the industry's worst-kept secret: high-quality human text is running out (the Villalobos et al. estimate put the usable stock of public human text at roughly 300T tokens, within reach of frontier runs), and synthetic data is what frontier labs feed their models now.

Everyday Analogy

Building a model is like training a doctor. Pre-training is all of school and medical school: years of reading everything, absorbing general knowledge at enormous cost. Supervised fine-tuning is residency: watching experts handle real cases and imitating them. Preference tuning is bedside-manner coaching: among correct answers, learning which ones patients actually find helpful. The newest stage, RL on verifiable problems, is board-exam drilling: solving thousands of problems where the answer can be checked, until the method itself improves. Each stage shapes different behavior, and costs a different amount: medical school takes years and a fortune; bedside coaching takes weeks.

The Pipeline, Stage by Stage

1. Data curation. Trillions of tokens are scraped, deduplicated, filtered for quality and toxicity, and mixed by a deliberate recipe. Llama 3's report describes a mix of roughly half general knowledge, with substantial slices of math/reasoning and code, because code and math data measurably improve reasoning even on non-code tasks. Quality classifiers (themselves models) grade every document; deduplication runs at document, paragraph, and near-duplicate levels. Data decisions here matter as much as architecture.
2. Pre-training. The expensive stage: weeks to months of GPU time learning next-token prediction (see AI-009) over the corpus. Llama 3.1 405B pre-trained on ~15.6T tokens using ~30.8M H100-GPU-hours; Sam Altman said GPT-4's training cost "more than $100 million." Scaling laws guide the budget split: DeepMind's Chinchilla result showed a compute-optimal ratio near 20 training tokens per parameter, though production models deliberately "overtrain" far beyond it (Llama 3 8B saw ~1,875 tokens per parameter) because inference cost rewards smaller models trained longer. Output: a base model, brilliant autocomplete with no instruction-following or manners. Ask it a question and it may continue with three more questions, because that's what web text does.
3. Supervised fine-tuning (SFT). Thousands to millions of curated instruction→response examples teach the base model to be an assistant: follow instructions, format answers, adopt a persona, refuse harmful requests. InstructGPT's original SFT set was only ~13K demonstrations, tiny next to 15T pre-training tokens, which is the clearest evidence that this stage steers rather than teaches.
4. Preference tuning (RLHF / DPO). Humans (or AI judges) compare pairs of responses; a reward model learns the preferences, and the LLM is optimized against it (classically via PPO), or the comparison data is used directly (DPO, which skips the separate reward model and RL loop entirely, cheaper and more stable, widely used in open models like Zephyr and Llama 3's pipeline alongside rejection sampling). This is where tone, judgment, helpfulness-vs-harmlessness balance, and much of a model's "personality" are shaped. The landmark InstructGPT result: users preferred the 1.3B RLHF-tuned model over the 175B base GPT-3. Alignment tuning beat a 100× parameter advantage on perceived quality.
5. RL for reasoning (the newest stage). For reasoning models (see AI-063), reinforcement learning on automatically checkable tasks (math answers that match, unit tests that pass) teaches genuine multi-step problem solving. Unlike RLHF's learned reward model, the reward here is a deterministic verifier, which resists reward hacking and scales without human labels.

A useful mental model of the economics: pre-training consumes upwards of 95% of the compute budget and produces the capability; everything after is comparatively cheap steering, and that is exactly why fine-tuning cannot add what pre-training never learned (see AI-051).

Worked Example: Budgeting a Pre-Training Run

Trace the arithmetic labs actually do, using Llama 3.1 405B's published numbers:

1. Compute required. The standard estimate is FLOPs ≈ 6 × N × D (six floating-point operations per parameter per token). For N = 405B parameters and D = 15.6T tokens: 6 × 4.05×10¹¹ × 1.56×10¹³ ≈ 3.8×10²⁵ FLOPs.
2. GPU-time. An H100 delivers on the order of 1×10¹⁵ dense BF16 FLOPs/s; real training achieves maybe 40% utilization (MFU), so ~4×10¹⁴ effective. 3.8×10²⁵ ÷ 4×10¹⁴ ≈ 9.5×10¹⁰ GPU-seconds ≈ 26M H100-hours, satisfyingly close to Meta's reported ~30.8M (their run included overhead, restarts, and annealing phases).
3. Dollars. At a $2/hour internal H100 cost, 30.8M hours ≈ $60M+ of compute for the pre-training run alone, before data, staff, ablations, and the post-training stages. (Compare DeepSeek-V3: an MoE with only 37B active parameters cut the same-scale bill to a reported ~$5.6M; the active-parameter term N is what the 6ND formula charges you for. See AI-064.)
4. Chinchilla check. Compute-optimal D for 405B parameters would be ~8T tokens (20 × 405B). Meta trained on nearly double that, deliberately overtrained, because a better-per-parameter model is cheaper to serve forever after.
5. Post-training, for contrast: SFT on ~1M examples averaging 1K tokens = 10⁹ tokens, a ~15,000× smaller data pass than pre-training. This ratio is the pipeline's economics in one number.

The Synthetic Data Turn

The internet's supply of high-quality human text is largely consumed; frontier runs at 15–30T tokens are within an order of magnitude of everything usable ever published. Frontier training now depends on synthetic data: text generated by models to train models:

• Instruction data: a strong model generates and answers thousands of instruction variants (self-instruct). Stanford's Alpaca (2023) made this famous by fine-tuning Llama on 52K GPT-generated instructions for under $600 of API cost, replacing what would have been months of human annotation.
• Reasoning traces: a reasoning model writes worked solutions; correct ones (verified by checking the final answer or running the tests) become training data. DeepSeek-R1's ~800K curated traces trained its whole distilled family (see AI-065 and AI-063).
• Textbook-style data: Microsoft's Phi series showed a small model trained on synthetic "textbook-quality" data can outperform models many times larger trained on raw web text. Phi-1, at 1.3B parameters, beat much larger models on code benchmarks. Quality beats quantity when you can manufacture quality.
• AI-judged preferences: models grade other models' outputs (RLAIF, Anthropic's constitutional AI approach), scaling preference data far beyond human labeling capacity: a human rater costs dollars per comparison; an AI judge costs a fraction of a cent.
• Filling distribution gaps: synthetic generation targets what the web lacks, including low-resource languages, rare API usage, safety edge cases, and long multi-turn tool-use conversations for agent training.

The Risk: Model Collapse and Its Management

Training repeatedly on unfiltered model output degrades quality: errors compound, diversity shrinks, and the tails of the distribution vanish first (rare facts, minority dialects, unusual styles). Shumailov et al.'s "curse of recursion" paper demonstrated the extreme case, dubbed model collapse: generations of models trained on their predecessors' outputs drift toward bland, low-variance text and eventually nonsense.

So why do frontier labs succeed with synthetic data anyway? Because production pipelines never do the naive thing. Three defenses, always combined:

1. Verification. Synthetic examples are kept only if they pass a check that does not depend on the generator: the math answer verifies, the code passes tests, the judge model (a different model) scores it highly, or execution feedback confirms it. Generation is cheap; the filter is the product.
2. Anchoring in human data. Synthetic data is mixed with, not substituted for, human text. Research on collapse shows accumulation (adding synthetic to real) behaves far better than replacement (training only on synthetic).
3. Quality classifiers and deduplication run on synthetic data just as they do on web scrapes: the same curation stage, applied to a new source.

The one-line rule: synthetic data works when generation is cheap and verification is reliable; it fails when you can't tell good outputs from bad ones cheaply. That's why math and code lead the synthetic revolution (perfect verifiers) while creative writing lags (no verifier).

Decision Framework: Which Stage Do You Intervene At?

You will likely never pre-train, but you will constantly choose where in the pipeline to solve a problem. Work up the stack from cheapest:

1. Model lacks context, not capability (your product docs, today's data) → prompting / RAG (see AI-021). Minutes to build, no training.
2. Model has the knowledge but wrong behavior (format, tone, domain style) → SFT / fine-tuning on hundreds to thousands of examples (see AI-051). Consider generating those examples synthetically from a stronger model, with verification, exactly as this lesson describes. Check the teacher model's license first (see AI-066).
3. Model behaves inconsistently and you can rank outputs ("this answer is better than that one") → preference tuning (DPO) on comparison pairs, practical on open models with a few thousand pairs.
4. Model fails at a verifiable skill (your DSL, your test suite) → RL on verifiable rewards, the frontier technique, now accessible on open models but the most engineering-intensive option.
5. Model fundamentally lacks the capability → change models. No amount of downstream tuning adds what pre-training never captured.

The corresponding data question at every level: can you verify quality automatically? If yes, synthetic data slashes your labeling cost 10–100×. If no, budget for humans, or build a judge you've validated against human ratings on a sample.

Real-World Showcase

• Llama 3 (2024) used earlier Llama models throughout its own pipeline: generating SFT data, filtering with model-based quality classifiers, and rejection-sampling its own outputs for preference tuning, a self-improvement loop documented in a frontier technical report.
• Phi series (Microsoft) built small models on curated synthetic "textbooks"; phi-1's 1.3B parameters competing with far larger code models validated quality-over-quantity and made "data quality is all you need" a design philosophy.
• DeepSeek-R1 (2025) turned one reasoning model's ~800K verified traces into training data for a whole family of smaller reasoning models (1.5B–70B), the clearest public demonstration that the pipeline's stages compose: RL creates a capability once, distillation copies it cheaply.
• Alpaca (2023): 52K synthetic instructions, <$600, a weekend project that cloned assistant behavior onto an open base model and kicked off the open fine-tuning boom.

Common Misconceptions

• "Capability comes from fine-tuning." Capabilities are laid down in pre-training (~95%+ of compute); later stages steer behavior. InstructGPT's 13K SFT examples against GPT-3's hundreds of billions of pre-training tokens is the canonical ratio.
• "Synthetic data inherently degrades models: model collapse proves it." Unfiltered, replacement-style synthetic training degrades; verified, mixed synthetic data is how Llama 3, Phi, and R1 were actually built. The verifier is the difference.
• "RLHF adds knowledge." It reweights behavior the model already has, making preferred response styles more likely. A model that doesn't know your API before RLHF still doesn't know it after.
• "More data is always better." Phi beat larger-corpus models with less, better data; deduplication (removing data) reliably improves models. The recipe, meaning quality and mixing ratios of code, math, and web, outranks raw volume.
• "The stages are independent." The mix is engineered end to end: labs add extra math and code to pre-training specifically so that RL for reasoning has something to amplify later.
• "Human data is obsolete." Human data anchors every stage; synthetic data extends it where verification is possible. Preference tuning in particular still leans on human judgment to calibrate the AI judges.

Try It Yourself

1. Run a miniature synthetic-data pipeline in any chatbot: (1) ask it to generate ten diverse customer-support questions for a fictional product; (2) ask it to answer each; (3) act as the verifier and grade each answer, keeping only the good ones. You now hold a small, filtered synthetic SFT dataset, produced exactly the way labs do it, minus the scale.
2. Feel the base-model difference: find a hosted base model (several open playgrounds label them) and give it "What is the capital of France?" Watch it complete rather than answer, perhaps continuing with more quiz questions. That gap between completing and answering is everything SFT does.
3. Redo the worked example's step 1 for a model you use: pick a parameter count and token count from any technical report and compute 6ND. Compare your figure against the reported GPU-hours and see how close the 40%-utilization assumption lands.

Mini Project

Build and evaluate a verified synthetic dataset. Pick a narrow skill with a checkable answer, say, converting natural-language date expressions into ISO 8601 ("next Friday", "three weeks from March 1st"). (1) Use a strong model to generate 100 input/output pairs. (2) Write a Python verifier that programmatically checks each output (parse the date, recompute it independently, compare). (3) Measure the rejection rate: what fraction of raw synthetic data failed verification? (4) Take a weaker model (small local model via Ollama) and measure its accuracy on 20 held-out cases with zero-shot prompting vs with 10 verified examples in the prompt. Report: rejection rate, accuracy before/after, and one paragraph on what this implies about scaling the same loop to fine-tuning. You will have run the generate→verify→filter→train loop that powers frontier pipelines, end to end, in an afternoon.

Key Takeaways

• Pipeline: curate → pre-train → SFT → preference tuning → (for reasoning models) RL on verifiable tasks; each stage shapes something different: knowledge, instruction-following, judgment, reasoning.
• The economics are wildly lopsided: pre-training consumes ~95%+ of compute (Llama 3.1 405B: ~3.8×10²⁵ FLOPs, ~30.8M H100-hours; GPT-4: reportedly >$100M), while SFT can be a ~15,000× smaller data pass, which is why later stages steer but cannot add capability.
• Scaling laws budget the run: Chinchilla's ~20 tokens/parameter is compute-optimal, but production models deliberately overtrain because inference cost rewards smaller models trained longer.
• Synthetic data now feeds every stage: self-instruct SFT (Alpaca's 52K examples for <$600), verified reasoning traces (R1's ~800K), textbook-style corpora (Phi), and AI-judged preferences (RLAIF).
• Model collapse is real but managed: verification, mixing with human data, and quality filtering are the three defenses; synthetic data works exactly where verification is cheap and reliable.
• When you need to change a model's behavior, walk up the stack: prompt/RAG, then SFT, then DPO, then RL, then switch models. Let "can I verify quality automatically?" decide whether your data is synthetic or human.