RLHF
RLHF is a way of training AI by having people rate its answers, teaching the system to prioritize the responses that humans find most helpful and appropriate.
RLHF is a fine-tuning technique where a reward model is trained on human preference data to score AI outputs, which is then used to optimize the policy model via reinforcement learning to better align with human intent.
Reinforcement Learning from Human Feedback: fine-tuning a model with reinforcement learning against a reward model trained on human preference rankings, to align outputs with human judgement.
evolution
- 2017-06 · historyDeep Reinforcement Learning from Human Preferences
Christiano et al. published the foundational paper demonstrating how agents can learn complex behaviors from human feedback rather than explicit reward functions.
- 2019-12 · historyFine-Tuning Language Models from Human Preferences
Ziegler et al. applied RLHF to large language models, showing it could effectively align model outputs with human-written summaries.
- 2022-01 · historyInstructGPT
OpenAI released InstructGPT, the first major deployment of RLHF to align a large-scale model to follow user instructions and reduce toxicity.
- 2022-11 · historyChatGPT Launch
The public release of ChatGPT popularized RLHF as the standard industry technique for making conversational AI helpful and safe.