TL;DR: Alignment is the work of training AI models to pursue what people actually intend, not just what they literally said. A model can be enormously capable and still be a failure if it optimizes the wrong thing - so alignment techniques like RLHF teach models human preferences directly, from examples of what people approve of and reject.
How it works
Tell a system to "maximize engagement" and it may learn that outrage and clickbait score best. It did exactly what it was told - and not at all what was meant. That gap between the literal objective and the human intent behind it is the alignment problem. A raw language model fresh out of pretraining has the same issue: it is trained only to predict the next token, so it will just as happily continue a harmful request as a helpful one. Alignment is the set of techniques that closes the gap.
The workhorse technique is RLHF - reinforcement learning from human feedback. Human raters compare pairs of model outputs and pick the better one; those judgments train a reward model that scores outputs automatically; the language model is then fine-tuned to produce outputs the reward model scores highly. Variations replace parts of the loop: direct preference optimization (DPO) skips the separate reward model, and constitutional approaches have the model critique and revise its own outputs against a written set of principles, reducing how much human labeling is needed.
Why is this hard? First, human values are fuzzy, contested, and context-dependent - there is no clean objective to optimize. Second, models learn to game the proxy: a reward model that favors agreeable answers produces sycophancy, where the model tells users what they want to hear rather than what is true. Third, behavior that looks aligned during training can fail in situations the training never covered. That is why alignment work leans so heavily on red-teaming and behavioral evals - the training is only as good as the testing that probes it. Alignment is one pillar of the broader AI safety effort, and the pillar that lives inside the model itself.
Where it sits in the AI stack
Alignment happens between pretraining and deployment - it is the stage that turns a raw text predictor into an assistant that follows instructions and declines harmful ones:
Key techniques and implementations
-
RLHF
Human preference rankings train a reward model that steers the language model toward approved behavior.
-
Constitutional AI
The model critiques and revises its own outputs against written principles, cutting the human labeling load.
-
DPO
Direct preference optimization trains on preference pairs without a separate reward model.
-
Behavioral evals
Test suites that probe refusals, sycophancy, and honesty to verify the training actually held.
Related entries
- AI safety The practice of preventing AI systems from causing harm, from everyday failures to large-scale misuse.
- RLHF (reinforcement learning from human feedback) Training a model toward answers humans prefer by scoring its outputs and reinforcing the good ones.
- Fine-tuning Further training a pretrained model on your own examples to specialize its behavior for a task.
- Responsible AI The organizational practice of building AI systems that are fair, transparent, and accountable.