Training
Training is the process of repeatedly showing a model examples, measuring how wrong its predictions are, and nudging millions of weights slightly in the direction that reduces the error, until the model is wrong as rarely as possible.
Learning Objectives
- Describe the training loop: predict, measure error, adjust, repeat.
- Explain loss as the "wrongness score" that training minimizes.
- Understand gradient descent through the fog-on-a-hill intuition.
- Recognize overfitting and why held-out test data is sacred.
- Connect training scale to why frontier models cost millions.
Why This Matters
Training is where all the capability comes from; a model is nothing but the residue of its training. Understanding the loop explains the vocabulary of the field (loss, epochs, gradient descent, overfitting), why training is astronomically expensive while using a model is cheap, and why "just retrain it" is never a casual fix.
Everyday Analogy
Learning to shoot basketball free throws: you shoot (predict), see the miss, too far left (measure error), adjust your aim slightly right (update), and shoot again. Ten thousand shots later your form has converged on something reliable. Nobody handed you equations for elbow angle; the feedback loop carved the skill into you. Training carves skill into weights the same way, one small correction at a time, millions of times.
The Training Loop
For a model learning to recognize cats:
- Predict. Show a photo. Untrained weights are random, so the model guesses: "dog, 62%."
- Measure the error (loss). The label says cat. A loss function converts the miss into a number, say 2.4. Bigger = more wrong.
- Compute the fix (gradients). Calculus determines, for each of the millions of weights, which direction of nudge would have made the error smaller.
- Update. Every weight moves a tiny step in its improving direction. The step size is the learning rate.
- Repeat, millions of times over the whole dataset (each full pass = one epoch), until the loss stops shrinking.
Early in training, loss plummets; later it inches down. Watching the loss curve fall is how practitioners literally see a model learning.
Gradient Descent: Fog on a Hill
You are standing on a foggy hillside and want to reach the valley floor. You cannot see the valley, but you can feel the slope beneath your feet. Strategy: step downhill, feel again, step again.
That is gradient descent. The landscape is the loss (altitude = wrongness) across all possible weight settings; the slope under your feet is the gradient; each step is one weight update. The learning rate is your stride: too small and the descent takes forever, too large and you overshoot the valley and bounce between hillsides. Modern training is this walk in a landscape with billions of dimensions, but the fog-walk intuition still holds.
Overfitting: Memorizing Instead of Learning
A student who memorizes past exam papers aces practice tests and fails the real exam. Models do exactly this: with enough capacity and training time, a network can memorize its training examples instead of learning the general pattern: perfect training accuracy, terrible real-world accuracy.
The defense is non-negotiable discipline: split your data.
| Split | Used for | Rule |
|---|---|---|
| Training set (~80%) | Adjusting weights | Model sees it constantly |
| Validation set (~10%) | Tuning decisions, spotting overfitting | Checked during training |
| Test set (~10%) | Final honest exam | Touched ONCE, at the end |
When training accuracy keeps rising while validation accuracy starts falling, that gap is overfitting, and it is the most-watched signal in applied ML.
The Price of Training
Why do headlines quote astronomical training costs? Scale each loop element up:
- Frontier LLMs train on trillions of tokens (AI-067 covers the data pipeline).
- Each update adjusts hundreds of billions of weights (AI-004).
- The loop runs on tens of thousands of GPUs for months.
GPT-4-class training runs are estimated at $50–100M+ in compute. This asymmetry between monstrous training and cheap prediction shapes the whole industry: few organizations train frontier models; everyone else fine-tunes (AI-051) or just calls them (AI-008).
Real-World Showcase
- Loss curves are mission control: at every AI lab, giant screens track the falling loss of runs costing thousands of dollars per hour. A spike in that curve pages an engineer at 3 a.m.
- Tesla's fleet learning: millions of cars send tricky driving frames back as new training data: the improve step of the loop, running planet-wide.
- DeepSeek-V3 made news by training a frontier-class model for ~$5.6M, an efficiency breakthrough measured entirely in training-loop economics (see AI-064).
Try It Yourself
- Play 20 rounds of "hot and cold": a friend picks a number 1–100, you guess, they say only "warmer/colder." Your shrinking guesses are gradient descent: error feedback steering updates.
- Overfit yourself: memorize these pairs: (2→7), (3→9), (5→13). Now predict for 4. If you said 11, you generalized the pattern (2x+3). If you could only answer for 2, 3, and 5, you memorized. Models face this exact fork.
- In TensorFlow Playground (browser), watch the loss number fall as the network trains, then add layers until the training loss is perfect but the test loss worsens. You just manufactured overfitting.
Common Mistakes
- Judging a model on training accuracy: only held-out test performance counts.
- Letting test data leak into training decisions, since the exam is void if the student saw the questions.
- Assuming training is a one-time event, when real systems retrain as the world drifts (AI-028 covers monitoring).
- Confusing training with inference. Training builds the model once at great cost; inference uses it cheaply forever after (next lesson).
Key Takeaways
- Training = predict → measure loss → compute gradients → nudge weights → repeat, millions of times.
- Gradient descent is a downhill walk through a foggy loss landscape; learning rate is stride length.
- Overfitting = memorizing the training set; held-out validation/test data is the detector.
- The loss curve is the heartbeat of every training run.
- Training is astronomically expensive; using the result is cheap. That gap is the economics behind the whole AI industry.
Glossary
- Training
- The loop of showing a model examples, measuring how wrong its predictions are, and nudging millions of weights slightly in the improving direction, repeated millions of times. Like basketball free throws, the feedback loop carves skill into the weights one small correction at a time.
- Loss
- The "wrongness score" a loss function assigns to each prediction: the model guesses "dog, 62%" for a cat and gets a loss of 2.4. Watching the loss curve fall is how practitioners literally see a model learning.
- Gradient Descent
- The core optimization strategy: like walking down a foggy hillside, feel the slope (gradient) under your feet and step downhill, repeatedly. The landscape is the loss across all possible weight settings, and each step is one weight update.
- Learning Rate
- The stride length of gradient descent. It sets how big each weight-update step is. Too small and training takes forever; too large and updates overshoot the valley and bounce between hillsides.
- Epoch
- One complete pass through the entire training dataset. Training typically runs many epochs until the loss stops shrinking.
- Overfitting
- Memorizing training examples instead of learning the general pattern: perfect training accuracy, terrible real-world accuracy, like a student who memorized past exam papers. The telltale signal is training accuracy rising while validation accuracy falls.
- Validation Set
- The ~10% data slice held out from weight updates and checked during training to tune decisions and detect overfitting. Distinct from the test set, which is touched exactly once at the end as the final honest exam.
- Test Set
- The held-out ~10% used once, at the very end, to measure honest real-world performance. If it leaks into training decisions, the exam is void: the student saw the questions.
References
Diagram
Knowledge Check
9 questions