TL;DR: RLHF - reinforcement learning from human feedback - trains a model toward answers people prefer. Humans rank pairs of model outputs, a reward model learns to predict those rankings, and the language model is then optimized to score well against it. It is the step that turned raw text predictors into helpful assistants.
How it works
A freshly pretrained LLM only continues text. Ask it "How do I write a resignation letter?" and it might respond with more questions, because forum threads in its training data look like that. Making it an assistant starts with supervised fine-tuning on example conversations - but you cannot write enough examples to cover everything, and many qualities you want (honesty, appropriate tone, refusing harmful requests gracefully) are far easier to recognize than to demonstrate. RLHF exploits that asymmetry: instead of showing the model perfect answers, you teach it to tell better from worse.
The pipeline has three stages. First, collect preferences: the model generates two or more responses to the same prompt, and human raters pick the better one, thousands of times. Second, train a reward model - a separate network that ingests a prompt and response and outputs a score predicting how a human would rate it. This is the crucial move, because the reward model can now judge unlimited new outputs without a human in the room. Third, optimize the language model with reinforcement learning (classically PPO) to produce responses the reward model scores highly, with a constraint that keeps it from drifting too far from its original abilities.
The approach has known failure modes. The model learns to maximize predicted approval, not truth - which can breed sycophancy (agreeing with the user), confident-sounding waffle, and reward hacking, where outputs exploit quirks of the reward model rather than getting better. Preference data also bakes in whose preferences were collected. These issues sit at the center of alignment work. Simpler successors now compete with the full pipeline: DPO trains directly on preference pairs without a separate reward model or RL loop, and RLAIF replaces human raters with an AI judge steered by written principles - cheaper, more scalable, and closely related to how LLM-as-judge evaluation works.
Whatever the variant, preference optimization is why chat models feel cooperative at all - and why their personalities differ: each lab's models reflect its own preference data and training choices.
Where it sits in the AI stack
RLHF is the final stage of post-training, between a pretrained model and the aligned assistant that ships:
Key tools and implementations
-
PPO
The reinforcement learning algorithm used in the classic RLHF pipeline.
-
DPO
Direct preference optimization - trains on preference pairs without a reward model or RL loop.
-
Constitutional AI / RLAIF
Replaces human raters with AI feedback guided by a written set of principles.
-
Hugging Face TRL
The open-source library for running preference training on open-weights models.
Related entries
- Fine-tuning Further training a pretrained model on your own examples to specialize its behavior for a task.
- Alignment Training AI models so their behavior matches human intent and values, not just the literal words of an instruction.
- LLM (large language model) A neural network trained on huge amounts of text to predict the next token, which lets it generate and understand language.
- LLM-as-judge Using a language model to score another model's outputs against a rubric instead of human review.