Reinforcement Learning & RLHF

Why This Matters

A raw pretrained LLM is not a chatbot. It is a next-token predictor that continues whatever text pattern it sees, which means a base model can just as easily complete a question with more questions, a rant, or a plausible-sounding wrong answer as with a helpful reply. The step that turned GPT-3 into ChatGPT, and every subsequent base model into a usable assistant, is alignment training, and RLHF is the technique that made it work at scale. Without understanding RL and RLHF, "fine-tuning" (AI-051) and "reasoning models" (AI-063) both look like unrelated magic instead of two applications of the same underlying training idea: use a reward signal to reshape a model's behavior after pretraining.

Everyday Analogy

Picture training a new barista. You could hand them a thick manual of exact rules ("always tamp with 30 pounds of pressure, always steam milk to 65°C"), which is what supervised learning does: memorize labeled correct answers. Or you could let them make drinks, have customers rate each one, and let the barista gradually adjust their technique toward whatever earns higher ratings, discovering on their own that leveling the tamp matters more than the manual ever specified. That is reinforcement learning: no one writes down the "correct" action for every situation, the learner improves by acting, observing a reward, and adjusting to get more reward next time. RLHF is that same barista training, except the customers rating each drink are human raters comparing two of the model's responses side by side.

Reinforcement Learning: The Core Loop

Reinforcement learning (RL) is a distinct branch of machine learning from the supervised learning covered in AI-002, where a model learns from labeled input-output pairs. RL has no labeled "correct answer" for every situation. Instead, it has four moving parts:

• Agent. The learner making decisions, a chess-playing program, a robot arm, or (in RLHF) the language model itself.
• Environment. Whatever the agent acts within, a chessboard, a warehouse floor, or (in RLHF) a conversation with a prompt.
• Action. A choice the agent makes, moving a chess piece, or generating the next token/response.
• Reward. A numeric signal telling the agent how good that action was, win/loss for chess, a human's preference rating for a chatbot response.

The agent's strategy for choosing actions is called its policy. Training adjusts the policy to maximize expected cumulative reward. Unlike supervised learning, where every example comes with the one right answer, RL only ever gets a scalar signal, good or bad, and must work out which of its own actions deserve the credit. This is why RL is notoriously sample-inefficient and unstable compared to supervised training, and why it took a specific breakthrough to make it work reliably for language models.

RL's most famous showcase predates LLMs entirely: DeepMind's AlphaGo and AlphaZero learned to play Go, chess, and shogi at superhuman level purely through self-play and a win/loss reward, no human game annotations required. The RLHF pipeline below applies the same trial-and-reward principle, just with a reward model standing in for the win/loss signal, since "was this response helpful" has no simple win condition to check automatically.

RLHF: The Three-Stage Pipeline

Reinforcement Learning from Human Feedback (RLHF) is the specific recipe that aligns a raw pretrained LLM into a helpful, instruction-following assistant. It runs in three stages, and this is the same pipeline briefly introduced in AI-051's fine-tuning lesson, expanded here in full:

Stage 1 — Supervised Fine-Tuning (SFT). Human writers produce high-quality example responses to a range of prompts, and the pretrained base model is fine-tuned on these (prompt, ideal response) pairs using ordinary supervised learning (AI-051). This gives the model a first, rough sense of the assistant format and tone, but it is expensive to scale by writing more examples, and it only teaches the model to imitate, not to improve past what the writers demonstrated.

Stage 2 — Reward Model Training. Human annotators are shown several responses to the same prompt (generated by the SFT model) and rank them from best to worst. This preference data trains a separate reward model: a model whose only job is to take a (prompt, response) pair and output a scalar score predicting how much a human would prefer that response. Critically, the reward model does not need to explain why a response is good, only to reproduce human relative judgments accurately enough to be useful as an automatic reward signal.

Stage 3 — RL Fine-Tuning with PPO. The SFT model (now called the policy) is fine-tuned further using Proximal Policy Optimization (PPO), a reinforcement learning algorithm. For each prompt, the policy generates a response, the reward model scores it, and PPO nudges the policy's parameters to make higher-scoring responses more likely, while a KL-divergence penalty keeps the policy from drifting too far from the original SFT model in any single update (which prevents it from "reward hacking" its way into degenerate outputs that fool the reward model but read as gibberish or repetitive text to a person).

This three-stage pipeline, SFT then reward model then PPO, is exactly what OpenAI described in the InstructGPT paper, and it is the same pipeline that turned GPT-3 into the first version of ChatGPT: a model that had memorized the internet but had no notion of "being helpful" until reward-model-guided RL reshaped its behavior toward it.

Constitutional AI: A Notable Variant

Anthropic's Constitutional AI (CAI) is a variant of this pipeline that reduces reliance on raw human labeling for harmlessness. Instead of only asking humans to rank responses, CAI has the model critique and revise its own outputs against a written set of principles (a "constitution"), generating its own preference data for a harmlessness reward model, then applying the same RL fine-tuning stage. The intent is the same as RLHF, shape behavior via a learned reward signal, but a portion of the preference labeling is done by an AI evaluating against explicit written principles instead of purely by human raters judging case by case.

DPO: RLHF Without a Separate Reward Model

Direct Preference Optimization (DPO), introduced by Rafailov et al. in 2023, is a lighter-weight alternative that has become popular for exactly this reason: it skips Stage 2 and Stage 3's separate reward model and PPO loop entirely. DPO reformulates the RLHF objective mathematically so that the same human preference data (pairs of "response A is better than response B") can be used to directly optimize the policy model with a simple classification-style loss, no reward model to train, no PPO's reinforcement-learning instability to manage, no separate model to keep in sync during training.

Many open-weight instruction-tuned models released since 2023 use DPO or a DPO variant instead of full PPO-based RLHF, precisely because it gets much of the alignment benefit at a fraction of the engineering complexity.

Why Reasoning Models Depend on This

AI-063 covers reasoning models, LLMs that spend extra "thinking" tokens working through a problem before answering. That capability is trained, not prompted, using reinforcement learning: DeepSeek-R1's training, for instance, used RL with automatically checkable rewards (did the final math answer match, did the generated code pass its tests) to reinforce token sequences that functioned as effective step-by-step reasoning, exploring, self-checking, and backtracking. There was no human writing out ideal chains of thought to imitate; RL selected for whichever reasoning patterns actually led to verifiably correct answers. In other words, reasoning models are what you get when you point the same reward-driven RL loop that RLHF uses for "be helpful" at a different, verifiable reward: "get the right final answer."

Real-World Showcase

• OpenAI's InstructGPT and ChatGPT are the canonical RLHF story: GPT-3's raw completions were reshaped by SFT, a reward model trained on human rankings, and PPO into a model that followed instructions and refused harmful requests, at a fraction of the parameter count of the larger, non-aligned GPT-3 that annotators nonetheless preferred.
• Anthropic's Constitutional AI trains Claude's harmlessness behavior partly through AI-generated critique-and-revision against written principles rather than purely human labels, reducing the volume of human review needed for safety-relevant preference data.
• DPO's rise across the open-weight ecosystem, models like Zephyr and many Llama and Mistral fine-tunes, shows the field converging on DPO as the practical default for teams that want RLHF-style alignment without operating a full PPO training loop.

Try It Yourself

1. Map a customer-support chatbot onto the RL loop: what is the agent, the environment, an action, and a plausible reward signal? Notice how much harder "reward" is to define than in a game like chess with a clear win/loss.
2. Sketch the three RLHF stages for a hypothetical company building an internal coding assistant: what would SFT examples look like, what would reward-model preference pairs look like, and what would the PPO step actually be optimizing?
3. Compare RLHF (PPO) and DPO on one axis each: engineering complexity, compute cost, and training stability. Which would you pick for a startup with limited ML infrastructure, and why?

Common Mistakes

• Treating "fine-tuning" and "RLHF" as the same thing. Fine-tuning (AI-051) can mean simple supervised training on curated examples; RLHF specifically means reward-model-guided reinforcement learning as its own additional stage.
• Assuming the reward model is perfectly aligned with true human preferences. Reward models are themselves approximations trained on limited preference data, and a policy over-optimized against a reward model's blind spots ("reward hacking") can produce outputs that score well but a human would not actually prefer.
• Believing RLHF adds new knowledge to the model. It reshapes behavior and style toward alignment goals; it does not teach new facts, that is what pretraining and RAG (AI-016) are for.
• Confusing DPO with "RLHF is dead." DPO achieves a similar alignment goal through a different, simpler optimization path; it is still fundamentally using human preference data to reshape the policy, just without a separate reward model and PPO loop.
• Assuming reasoning models "reason" the way RLHF assistants are "helpful." Reasoning-model RL rewards verifiable correctness (right answer, passing tests); RLHF rewards subjective human preference. Different reward signals, same underlying RL machinery.

Key Takeaways

• Reinforcement learning trains an agent to maximize reward through trial and error, with no labeled correct answer for every situation, unlike the supervised learning in AI-002.
• RLHF aligns a pretrained LLM in three stages: supervised fine-tuning, reward model training on human preference rankings, then PPO-based RL fine-tuning against that reward model.
• RLHF is what turned GPT-3 into InstructGPT and ChatGPT; Anthropic's Constitutional AI is a notable variant that uses AI-generated critiques against written principles for part of the preference data.
• DPO achieves similar alignment results as RLHF using preference data directly, without training a separate reward model or running PPO, and has become the lighter-weight default for many open-weight models.
• Reasoning models (AI-063) use the same RL machinery, but reward verifiable correctness in math and code rather than subjective human preference, which is why their "thinking" behavior emerges from training rather than prompting.