[NeurIPS 2023] RLHF and RLAIF Research Trends and Noteworthy Papers

Recently, large language models (LLMs) trained on massive amounts of data have shown remarkable performance gains. However, supervised fine-tuning methods that require direct expert datasets are limited in their ability to improve performance. To overcome this, many studies have been conducted on LLM reinforcement learning methodologies utilizing human or AI feedback, and various research results were introduced at NeurIPS (Neural Information Processing Systems) 2023. This post will introduce the research trends in Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF).

Recently, Reinforcement Learning from Human Feedback (RLHF)[1] has become a popular method of fine-tuning LLMs with reinforcement learning. The existing RLHF methodology goes through two standardized processes as follows.

1. 1. Reward Model Learning Process

   Reward model learning uses a given pairwise dataset (preference data, which ranks two answers to a question and people’s preferences for them) 

2. 2. Policy Learning Process

   LLM learning uses the learned reward model through RL algorithms such as Proximal Policy Optimization (PPO)^[2]

However, these RLHF methodologies must use two separate learning processes, learning from a single reward model representing human preferences, resulting in unstable learning and not easily improving performance. At NeurIPS 2023, various studies related to RLHF or RLAIF were presented to solve the aforementioned problems, and among them, the following two studies received significant attention.

1. Direct Preference Optimization: Your Language Model Is Secretly a Reward Model^[3]

Figure 1. LLM learning process through RLHF and DPO

This paper, published by Stanford University, proposes a methodology that can learn policies directly from preference data, deviating from the existing RLHF method, which learns a reward model from the preference data of a given human or AI and then learns a policy based on it. In this paper, the reward model and optimal policy obtained through the two-step learning process of the existing RLHF were transformed into a single optimization problem that can obtain an equivalent optimal policy under a mathematical proof, and a new objective function was proposed.

Figure 2. Comparing results of LLM learning using DPO and existing methodologies
(Left: Sentiment Generation Task results, Right: Summarization Task results)

To show that the direct preference optimization (DPO) proposed in this paper is effective compared to the existing RLHF methodologies, comparative experiments were conducted on benchmarks from various NLP domains such as controlled sentiment generation, summarization, and dialogue. This paper showed that the proposed DPO had stable learning behavior with higher performance compared to the traditional RLHF methodology on the given task.

2. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training^[4]

This paper, published by the University of Washington, proposed a methodology and benchmark to improve the performance of LLM by more clearly defining criteria and improving the existing preference-based RLHF methodology, which is learned solely using information about people’s preferences for answers.

Figure 3. Comparing the human feedback collection process and LLM learning method of the existing 
preference-based RLHF methodology and the newly proposed fine-grained RLHF methodology

Existing preference-based RLHF only receives information about a person’s preferences as feedback. Since different people have different baseline preferences, it is difficult to get meaningful learning signals from multiple people’s feedback. To address these issues and enable more effective learning, this paper shows that by clearly defining the metrics for which performance improvement is needed and utilizing fine-grained human feedback for learning, more effective learning can be achieved for a variety of metrics.

Figure 4. Performance results (left) and learning graph (right) of LLM improved through the fine-grained RLHF methodology

Figure 5. Human evaluation results of LLM performance improved through fine-grained RLHF methodology

To show that the fine-grained RLHF methodology proposed in this paper is effective compared to the existing preference-based RLHF methodology, comparative experiments were conducted on detoxification and long-form question-answering (QA) tasks. This paper shows that on these tasks, the proposed fine-grained RLHF can learn more effectively than the existing preference-based RLHF.

In addition to the papers discussed in this post, various studies on RLHF and RLAIF technologies using human and AI feedback were presented at NeurIPS 2023. In particular, the learning method using fine-grained feedback is expected to improve LLM performance in more diverse and specific aspects in the future. LG AI Research also considers LLM fine-tuning through RLHF and RLAIF as one of its important research topics and plans to research RLHF and RLAIF methodologies that can generate more suitable answers for users with different preferences by using feedback signals.

▶ Want to learn more about the NeurIPS 2023 Research Blog series? (Link)