Reinforcement learning (RL)

Reinforcement learning (RL) is a machine-learning paradigm where an agent learns by trial and error: taking actions, receiving rewards or penalties, and updating its behaviour to maximise cumulative reward. In classic RL, an agent learns to play chess or control a robot arm. In LLM training, the 'actions' are token choices and the 'rewards' come from human or AI evaluators who judge the quality of the model's outputs.

RL in everyday language

Think of RL like training a dog. You don't give the dog a manual of rules - you reward good behaviour and don't reward (or penalise) bad behaviour. Over thousands of repetitions, the dog figures out the policy that gets the most treats. LLM training with RL works the same way: the model generates many candidate responses, a reward signal scores each one, and the model's weights are updated to make high-scoring responses more likely in future.

RLHF: the four-step process

RLHF (Reinforcement Learning from Human Feedback) is the standard approach used by OpenAI, Anthropic, and Google to turn raw pre-trained models into helpful assistants:

1. Pre-train. Train the base model on vast text data to predict the next token. At this stage it's a capable but unpredictable text predictor.
2. Supervised fine-tuning (SFT). Fine-tune on curated examples of helpful, harmless conversations to establish a baseline of good behaviour.
3. Train a reward model. Human raters compare pairs of model outputs and pick the better one. A separate reward model is trained to predict these human preferences, creating a scalable proxy for human judgment.
4. RL optimisation. Use PPO (Proximal Policy Optimisation) to update the LLM's weights so it produces responses the reward model scores highly. Repeat thousands of times.

Without RLHF, a base model produces raw text completions - useful for researchers, but unpredictable for everyday users. RLHF is what makes models actually chat helpfully.

RLHF vs RLAIF vs DPO

RL for reasoning models

DeepSeek R1 demonstrated in early 2025 that RL alone - applied to a capable base model with outcome-based rewards (is the answer correct?) rather than human preference labels - could teach a model to generate long chains of thought spontaneously. The model learned to 'think' not because it was shown examples of thinking, but because thinking led to better outcomes that were rewarded. This showed that reasoning behaviour can emerge from RL without explicit supervision, and at a fraction of the compute cost of Western frontier models. It is now the dominant approach for training reasoning models across the industry.