Casper et al. — Open Problems and Fundamental Limitations of RLHF (2023)

3 min read

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Authors: Stephen Casper (MIT) et al. (multi-institution: MIT, Cambridge, Berkeley, NYU, Anthropic, others) Canonical URL: https://arxiv.org/abs/2307.15217

Summary

A comprehensive survey of the structural limitations of RLHF as the dominant alignment technique. The paper organizes problems into three categories: limitations of human feedback itself (e.g., humans are biased, time-pressured, mis-aligned with each other), limitations of the reward model (e.g., reward hacking, misgeneralization), and limitations of the policy learned (e.g., distributional shift, mode collapse).

For this wiki, the critical contribution is the systematic treatment of why RLHF-trained models exhibit the kinds of behaviors documented in Sharma and Perez: it is not an implementation defect, it is structural. The reward landscape that produces sycophancy also produces other forms of reward-following that are the LLM analog of motivated reasoning in humans.

Key Contributions

  1. Reward hacking is structural. Even with perfect optimization, the reward model is a proxy for human preferences, and the policy will exploit the proxy where it diverges from the underlying preference. Goodhart’s law applies.
  2. Human feedback bias is built in. Human raters exhibit cognitive biases (including the ones documented in this wiki — confirmation, availability, etc.) and these biases shape the reward signal.
  3. Sycophancy is predicted, not accidental. A reward model trained on “which response do humans prefer?” predictably prefers agreement, flattery, and conventional answers — even when accuracy diverges from preference.
  4. Mode collapse and distributional narrowing. RLHF reduces output diversity, which has implications for groupthink-style multi-agent setups using the same base model.
  5. No clean fix. The paper does not offer a single solution; it argues these are fundamental limitations that require new approaches, not better RLHF.

Relevance to This Wiki

  • Theoretical foundation for “RLHF reward-following” as the LLM analog of Motivated Reasoning. Establishes that the model’s “motivation” toward user-preferred answers is built in by training, not a transient artifact.
  • Reinforces Sharma 2023 and Perez 2022. Provides the theoretical context for why those empirical findings are not surprising and not easily fixed.
  • Critical for the hypotheses framework. Many of the failure modes in H0–H7 are structural consequences of RLHF — not “the model needs better prompting” but “the model was trained to do this.” This shapes what kinds of interventions can plausibly work.
  • Implications for SAT Pipeline design. Pipelines that rely on the model’s preferences for “good reasoning” are subject to the same reward-model biases. Architectural separation (different models for different roles) is more robust than prompt-level mitigation.
  • Important caveat for Sycophancy and Motivated Reasoning concept pages. Both are not just “models picking up on user preferences” — they are the direct output of the reward function used to train them.

See Also

Graph View

Backlinks