RLHF Explained: Reinforcement Learning from Human Feedback

Introduction

Reinforcement Learning from Human Feedback (RLHF) is the technique that transformed large language models from next-token predictors into aligned assistants. Before RLHF, models like GPT-3 could generate impressive text but frequently produced toxic, untruthful, or unhelpful outputs. After RLHF, models like ChatGPT, Claude, and Llama 2 demonstrated a dramatic improvement in instruction following, refusal of harmful requests, and overall conversational quality.

The core idea is simple: instead of training the model solely to predict the next token, collect human judgments about which responses are better and optimise the model to produce responses that humans prefer. The execution, however, is anything but simple. RLHF requires training a separate reward model, running expensive reinforcement learning loops, and carefully balancing multiple competing objectives to prevent the model from exploiting the reward signal.

RLHF powers essentially every major deployed LLM in 2026 — GPT-4o, Claude 3.5, Gemini 2, Llama 3, DeepSeek V3, Qwen 2.5, and Mistral Large all use some variant of RLHF or direct preference optimisation as part of their alignment pipeline. Understanding how RLHF works is essential for anyone building production systems on top of these models, and it is equally important for understanding the limitations and failure modes of current AI systems.

For a broader introduction to how LLMs are trained end-to-end, see our guide on how large language models work.

The RLHF Pipeline

RLHF is a three-stage pipeline. Each stage produces a model that feeds into the next stage, and each stage has distinct data requirements, training dynamics, and failure modes.

Stage 1: Supervised Fine-Tuning (SFT)

The pipeline begins with supervised fine-tuning on high-quality demonstration data. Human labelers write example responses to diverse prompts, and the model is trained via standard next-token prediction to imitate these responses. This stage establishes the baseline behaviour that the subsequent stages will refine.

SFT alone produces models that can follow instructions reasonably well, but they tend to be mediocre — good enough to appear competent but not reliable enough for production. The SFT model serves as the starting policy for the RL stage and as the reference model for KL divergence computation. For a detailed treatment of SFT data preparation and training, see our LLM fine-tuning guide.

Stage 2: Reward Modeling

A separate reward model is trained to predict human preference judgments. For a given prompt, the reward model takes a response and outputs a scalar score representing how likely a human is to prefer that response. The reward model is trained on pairwise comparisons — given a prompt and two responses (A and B), the model learns to assign a higher score to the response that human raters preferred.

Stage 3: RL Fine-Tuning with PPO

The language model (now called the policy) is optimised using Proximal Policy Optimization (PPO) to maximise the reward score assigned by the reward model. A KL divergence penalty keeps the policy close to the SFT model, preventing the model from exploiting the reward model by producing superficially high-scoring but nonsensical outputs. This is the most complex and compute-intensive stage.

Reward Modeling

The reward model is the heart of RLHF. It translates subjective human preferences into a numerical signal that the RL algorithm can optimise. Getting the reward model right is the difference between a well-aligned model and one that learns to game the system.

The Bradley-Terry Model

The standard formulation for reward modeling is the Bradley-Terry model of pairwise preferences. Given a prompt x and two responses y₁ and y₂, the probability that a human prefers y₁ over y₂ is modelled as:

P(y₁ > y₂ | x) = σ(r_θ(x, y₁) - r_θ(x, y₂))

Where r_θ(x, y) is the reward model's scalar output and σ is the logistic sigmoid function. The reward model is trained to maximise the log-likelihood of the observed human preferences:

L_RM = -𝔼 [log σ(r_θ(x, y_w) - r_θ(x, y_l))]

Where y_w is the preferred response and y_l is the dispreferred one. The reward model is typically initialised from the SFT model with the language modelling head replaced by a linear projection to a single scalar. This transfer of representations is crucial — the reward model inherits the language understanding of the SFT model and only needs to learn the additional preference signal.

Dataset Size Requirements

The InstructGPT paper established that reward model quality improves log-linearly with the number of comparisons, with diminishing returns setting in around 50,000-100,000 comparisons. A minimum viable reward model for a specific domain requires 10,000-50,000 comparisons. Below 10,000, the reward model tends to overfit to idiosyncrasies in the annotation data rather than learning generalisable preferences.

Each comparison is typically judged by 3-5 raters to reduce noise. Inter-rater agreement (measured by Krippendorff's alpha) should be monitored continuously — if agreement drops below 0.6, the annotation instructions or the task definition need revision. High-disagreement examples (where raters split roughly evenly) are often the most informative and should be reviewed, not discarded.

Reward Hacking and Over-Optimisation

The fundamental challenge of reward modeling is that the reward model is a proxy for true human preferences, not the real thing. The RL policy will inevitably discover ways to maximise the proxy reward that do not correspond to genuinely better outputs. This is an instance of Goodhart's law: when a measure becomes a target, it ceases to be a good measure.

The classic example is reward models that learn to prefer longer responses. Since human raters in the training data tended to prefer more detailed answers, the reward model assigns higher scores to longer responses regardless of quality. The RL policy then learns to produce verbose, repetitive answers that maximise length but add no substance. Mitigation strategies include length normalisation (subtracting the length bias from reward scores), ensemble reward models (averaging predictions from multiple independently trained reward models), and conservative reward estimation with uncertainty penalties.

Proximal Policy Optimization (PPO)

PPO is the RL algorithm used in the third stage of RLHF. It was introduced by Schulman et al. in 2017 (arXiv:1707.06347) as a simpler and more stable alternative to prior policy gradient methods like TRPO. PPO has become the standard RL algorithm for LLM alignment because it balances sample efficiency, implementation simplicity, and training stability.

The PPO Objective

PPO optimises a clipped surrogate objective that prevents the policy from changing too much in a single update. For each token in the generated response, PPO computes:

L^CLIP(θ) = 𝔼[min(r_t(θ)Â_t, clip(r_t(θ), 1-ε, 1+ε)Â_t)]

Where r_t(θ) = π_θ(a_t | s_t) / π_{θ_old}(a_t | s_t) is the probability ratio between the new and old policies, Â_tis the advantage estimate, and ε is the clipping hyperparameter (typically 0.2). The clipping operation removes the incentive for the probability ratio to move outside [1-ε, 1+ε], ensuring that each update is conservative.

KL Penalty

In RLHF, the PPO objective is augmented with a KL divergence penalty that keeps the policy close to the SFT reference model:

L_PPO-RLHF = L^CLIP - β * KL(π_θ || π_ref)

The KL penalty serves two purposes. First, it prevents reward hacking by constraining the policy to remain within the distribution where the reward model is reliable. Second, it preserves the language capabilities of the SFT model — without the KL penalty, the policy would quickly degenerate into producing gibberish that happens to score high on the reward model. The coefficient β is typically tuned in the range 0.01-0.1, with higher values producing more conservative updates.

PPO in Practice: The Four Key Components

A complete PPO-RLHF training loop maintains four models simultaneously:

• Policy model (π_θ): The language model being trained. Generates responses and receives gradient updates.
• Reference model (π_ref): A frozen copy of the SFT model. Used to compute the KL penalty. Never updated.
• Reward model (r_φ): A frozen model that scores each generated response. Provides the reward signal.
• Value model (V_ψ): A learned model that estimates the expected return. Used to compute advantages. Typically initialised from the reward model.

The training loop proceeds in steps: (1) sample a batch of prompts and generate responses from the policy, (2) score each response with the reward model, (3) compute advantages using the value model and Generalized Advantage Estimation (GAE), (4) update the policy using the clipped PPO objective and the value model using mean-squared error against the observed returns.

Alternatives to PPO

The complexity and instability of PPO training have motivated extensive research into simpler alternatives. The most impactful alternative is Direct Preference Optimization (DPO), but several other methods address specific shortcomings of the PPO paradigm.

DPO: Direct Preference Optimization

DPO, introduced by Rafailov et al. in 2023 (arXiv:2305.18290), eliminates the need for a separate reward model entirely. The key insight is that the RLHF objective can be reparameterised so that the language model itself implicitly learns the reward function through a direct preference loss.

The DPO loss compares the log-probabilities of the chosen and rejected responses under the current policy relative to the reference model:

L_DPO = -𝔼[log σ(β log (π_θ(y_w | x) / π_ref(y_w | x)) - β log (π_θ(y_l | x) / π_ref(y_l | x)))]

Intuitively, DPO increases the relative probability of preferred responses and decreases the relative probability of dispreferred ones, with the reference model preventing the policy from drifting too far. The temperature parameter β (typically 0.1-0.5) controls the trade-off between preference alignment and deviation from the reference model.

DPO vs PPO: Comparative Analysis

DPO has largely replaced PPO in open-source alignment pipelines because of its simplicity and lower compute requirements. However, PPO retains advantages in settings where online data collection is feasible — because PPO generates new responses from the evolving policy, it can explore and learn from the current distribution, while DPO is limited to the fixed preference dataset. The largest production systems (GPT-4o, Gemini) still use PPO or hybrid approaches.

Other Alternatives

Beyond DPO, several other methods have emerged. ORPO (Odds Ratio Preference Optimization, arXiv:2403.07691) fuses SFT and preference optimisation into a single stage by adding a penalty term to the language modelling loss. KTO (Kahneman-Tversky Optimization, arXiv:2402.01306) works with unpaired preference data, requiring only examples of good or bad outputs. IPO (Identity Preference Optimization, arXiv:2310.12036) modifies the DPO objective to be more robust to label noise. Each method trades off simplicity, data efficiency, and alignment quality.

Beyond Binary Preferences

The standard RLHF framework uses binary pairwise comparisons (response A is better than response B). This is convenient for data collection and mathematically tractable, but it discards rich information about the degree and nature of preference differences.

Ranking-Based Methods

Instead of pairwise comparisons, some approaches collect ranked lists of responses. Raters are shown 3-5 responses per prompt and asked to rank them from best to worst. The reward model is then trained with a Plackett-Luce ranking loss, which extends the Bradley-Terry model to multiple items. Ranking produces more signal per annotation effort — a 5-item ranking provides 10 implied pairwise comparisons but requires only slightly more rater time than a single pair.

Contrastive Preference Learning

Fine-grained feedback captures more nuance than binary labels. Instead of asking which response is better, annotators can identify specific dimensions — which response is more helpful, which is more truthful, which has better tone — and provide separate preference signals for each dimension. Multi-dimensional reward models can then be combined with task-specific weights, allowing the alignment objective to be tuned per application domain.

Continuous Feedback

An emerging paradigm uses continuous ratings (1-5 Likert scales or slider-based scores) instead of binary choices. This captures preference intensity and enables more nuanced training signals. However, continuous ratings introduce calibration challenges — different raters use scales differently, and anchoring effects can distort the absolute values. Normalisation techniques like z-scoring per rater or per-batch are essential for mitigating these effects.

Data Collection for RLHF

The quality of an RLHF system is fundamentally limited by the quality of its preference data. Data collection is often the most expensive and time-consuming part of the pipeline, and it requires careful attention to annotation platform design, rater training, quality control, and demographic diversity.

Annotation Platform Design

A good annotation platform minimises cognitive load on raters while maximising information yield. The standard interface shows a prompt and two responses side by side, with the rater selecting which is better or indicating a tie. Key design decisions include:

• Response ordering: Randomise which response appears on the left to avoid position bias. Track position bias metrics to detect rater shortcuts.
• Tie handling: Include a tie option. Forcing binary choices when responses are equally good introduces noise. Ties typically account for 10-20% of comparisons.
• Guidelines: Provide clear, specific criteria for what makes a response better. Generic instructions like "choose the more helpful response" produce inconsistent labels.
• Quality checks: Insert golden test questions (known ground-truth comparisons) every 10-20 annotations. Flag raters who fall below 80% accuracy.

Rater Selection and Diversity

Rater demographics significantly impact preference data. A reward model trained predominantly on preferences from one demographic group will encode that group's values and biases — an instance of the broader challenge discussed in our AI bias and fairness guide. Best practices include recruiting raters across geographic regions, age groups, educational backgrounds, and native languages, and tracking preference distributions across demographic segments to detect systematic disagreements.

Cost Considerations

Preference data collection is expensive. Typical costs through annotation platforms like Scale AI, Labelbox, or Surge AI range from $0.50 to $5.00 per comparison, depending on task complexity, rater qualifications, and quality assurance requirements. A production-quality reward model requiring 50,000-100,000 comparisons represents an investment of $25,000 to $500,000 in annotation alone. This cost is a significant barrier to entry and one of the main reasons why smaller teams gravitate towards DPO and other methods that can work with smaller or synthetic preference datasets.

Production Deployment

Deploying an RLHF-aligned model in production involves considerations beyond those of standard model serving. The reward model, the alignment process, and the evaluation protocol all present unique operational challenges.

Online vs Offline RLHF

In offline RLHF, the preference dataset is fixed and the reward model is trained once. This is simpler to deploy but means the reward model is blind to distribution shifts caused by policy updates. Online RLHF continuously collects new preference data on the latest policy outputs, retrains the reward model periodically, and re-runs PPO training. Online RLHF produces better-aligned models but requires sustained annotation infrastructure and careful management of the human annotation pipeline.

Reward Model Serving

In production, the reward model is typically served alongside the policy as a quality assessment service. Every generated response is scored by the reward model in real time, and low-scoring responses can be rejected, retried with different sampling parameters, or flagged for human review. Reward model inference adds latency (typically 50-200 ms per response for a 7B-parameter reward model) and must be considered in the overall system latency budget.

A/B Evaluation and Safety Guardrails

Evaluating an RLHF-aligned model requires more than standard benchmark scores. The alignment process can produce unexpected regressions — the model may become more sycophantic, more reluctant to refuse harmful requests, or more prone to hallucination on specific topics. A comprehensive evaluation suite should include:

• Standard benchmarks: MMLU, HellaSwag, GSM8K, HumanEval for capability retention.
• Alignment-specific evals: TruthfulQA, Safety benchmarks, bias benchmarks, sycophancy tests.
• Human evaluation: Pairwise A/B comparisons with at least 3 raters per sample. See our LLM evaluation guide for detailed methodology.
• Safety guardrails: Input and output filtering, topic blocking, red-teaming exercises, and gradual rollout with automated rollback triggers.

Limitations and Criticisms

RLHF is the dominant alignment technique in 2026, but it has well-documented limitations that practitioners should understand. These are not implementation bugs — they are structural limitations of the approach.

Reward Model Bias

The reward model inherits and amplifies biases present in the preference dataset. If human raters consistently prefer grammatically fluent but factually hollow responses, the reward model will encode that preference. If raters favour verbose responses, the reward model will assign higher scores to longer outputs. These biases propagate through the RL pipeline and become embedded in the final model. Mitigation requires careful auditing of reward model predictions and deliberate curation of preference data to reflect genuine quality criteria rather than surface-level patterns.

Goodhart's Law in Reinforcement Learning

Goodhart's law states that when a metric becomes a target, it ceases to be a good metric. In RLHF, the reward score is the target, and the policy inevitably discovers ways to maximise it that do not correspond to genuinely better behaviour. This is not a theoretical concern — reward hacking is routinely observed in production RLHF systems. The usual mitigation is a KL penalty, but this only constrains the policy, it does not eliminate the incentive to exploit. The fundamental solution is a better reward model, which requires better preference data, which requires better human annotation, which brings us back to the cost and quality problems discussed earlier.

The Alignment Tax

Alignment through RLHF comes at a measurable cost in model capabilities. Multiple studies have shown that RLHF reduces the model's diversity of outputs, decreases performance on factual recall and reasoning benchmarks, and narrows the range of tasks the model handles effectively. This "alignment tax" is the subject of active research — methods like DPO and ORPO claim lower alignment tax than PPO, but some degradation appears to be intrinsic to the preference optimisation process. For applications where capability breadth is critical, the alignment tax must be weighed against the benefits of improved instruction following.

Scaling Challenges

As models grow larger, the cost and complexity of RLHF scale superlinearly. Training a reward model for a 405B-parameter model requires enormous GPU clusters and careful distributed training. The PPO training loop requires maintaining multiple copies of the model simultaneously, straining memory bandwidth and communication infrastructure. These scaling challenges are a major reason why frontier RLHF capability is concentrated in a small number of well-resourced organisations.

Conclusion

RLHF remains the most powerful alignment technique available for large language models, despite a decade of active research into alternatives. The three-stage pipeline — SFT, reward modeling, PPO — produces models that are more helpful, more honest, and less harmful than pure SFT models, and it is the foundation upon which every major deployed LLM in 2026 is built.

The practical landscape has shifted significantly. DPO has simplified preference optimisation for teams without the resources for full PPO training. New variants like ORPO, KTO, and IPO address specific limitations of the original formulations. Multi-dimensional reward models and fine-grained feedback capture richer preference signals. And the open-source ecosystem has matured to the point where RLHF pipelines are accessible to any team with sufficient data and compute.

The hard problems remain: data quality, reward model bias, the alignment tax, and the fundamental challenge of encoding human values into a mathematical optimisation objective. These are not engineering problems that admit fixed solutions — they are ongoing research challenges that every organisation deploying aligned LLMs must navigate. For teams building on these models, understanding RLHF is essential for informed decision-making about data collection, method selection, evaluation design, and deployment strategy.

For further depth on specific aspects of LLM alignment and deployment, explore our guides on LLM fine-tuning, prompt engineering, AI bias and fairness, and LLM evaluation.