[New Zhiyuan Guide]After ChatGPT was born, RLHF became the focus of researchers. Google’s latest research suggests that AI can achieve the same effect as RLHF after labeling preferences without human labeling.
If the “human” in RLHF is replaced, is it feasible?
The latest research by the Google team proposes to replace humans with large models for preference labeling, which is AI Feedback Intensive Learning (RLAIF).
It was found that RLAIF can produce comparable improvements to RLHF with a 50% win rate without relying on human annotators.
At the same time, Google research once again proved that RLAIF and RLHF have a winning rate of more than 70% compared with supervised fine-tuning (SFT).

A key part of training large language models today is RLHF. Humans make the responses more useful by rating the quality of the AI’s output.
However, this requires a lot of effort, including exposing many annotators to the harmful content of the AI output.
Now that RLAIF is comparable to RLHF, future models do not require human feedback and can also be improved through self-loops.

RLHF doesn’t need humans anymore
Currently, RLHF has become the core method for fine-tuning large models, including ChatGPT, Bard and other models that adopt this paradigm.
Specifically, RLHF is divided into three steps: pre-training a supervised fine-tuning LLM; collecting data to train a reward model; fine-tuning the model with RL.

With RLHF, large models can be optimized for complex sequence-level objectives that are difficult to distinguish with conventional SFT.
However, a very real question is whether RLHF needs large-scale high-quality human-annotated data, and whether these data can achieve a superior result.
Prior to this Google study, Anthropic researchers were the first to explore using AI preferences to train RL fine-tuned reward models.
They proposed RLAIF in “Constitutional AI” for the first time, and found that LLM was highly consistent with human judgment, and even performed better than humans in some tasks.

However, this study did not compare human and artificial intelligence feedback, so whether RLAIF can replace RLHF has not yet been definitively answered.
Google’s latest research is mainly to solve this problem.
The researchers directly compared RLAIF and RLHF on the model summarization task.
Given 1 text and 2 candidate answers, use off-the-shelf LLM to give a preference label.
Then, a reward model (RM) is trained based on LLM preferences and contrastive losses. Finally, the policy model is fine-tuned through reinforcement learning, and the reward model is used to give rewards.

So, what is the difference between the RLAIF approach proposed by Google and Anthropic?

Google itself explains in the text,
– Google: Train a reward model based on AI-annotated preferences, then do RL fine-tuning.
– Constitutional AI: Improve supervised learning models by iteratively asking the LLM to generate better responses according to the constitution.

AI self-labeling, self-improvement
What is the process of the RLAIF method proposed by Google in the latest research?
Large language model for preference labeling
Researchers use “off-the-shelf” LLMs to annotate preferences between two candidates.
This is a model pre-trained or instruction-tuned for general use, but not fine-tuned for a specific downstream task. Given a piece of text and two candidate abstracts, LLM is asked to rate which abstract is better.The input structure of LLM is as follows:
1. Preamble
Instructions that introduce and describe the task at hand
2. Multiple sample instances (optional)
A piece of text, a pair of abstracts, a rationale for an idea, and a preference judgment
3. Samples to label
A piece of text and a pair of summaries to annotate
4. Ending
Prompt end string for LLM (e.g. “Preferred Summary=”)

After providing input to the LLM, the researchers obtain the log probabilities of generating tokens “1” and “2” and calculate a softmax to obtain the preference distribution.
There are many ways to obtain preference annotations from LLM, such as decoding free-form responses from the model and extracting preferences heuristically (eg: output = “the first summary is better”), or representing the preference distribution as a one-hot representation (one -hot representation). However, the researchers did not try these alternatives because their method already produced high levels of accuracy.
The researchers experimented with two types of preambles: the first, “Base,” which simply asks “which summary is better?”, and the second, “OpenAI,” which mimics the approach used to generate the OpenAI TL;DR preference dataset. A rating directive for the human preference tagger, with details about what constitutes a strong summary. As shown below.

The researchers also experimented with contextual learning by adding a small number of samples to the prompts, where samples were manually selected to cover different topics. Solve the positional deviation.
Previous findings have shown that the order in which candidates are presented to the LLM may affect which candidate the LLM judges to prefer. The researchers found evidence of this positional bias, especially for smaller-sized annotated LLMs.

To mitigate positional bias in preference labeling, we run two inferences on each pair of candidates, with the order in which the candidates are submitted to the LLM reversed. The results of the two inferences are then averaged to obtain the final preference distribution.
chain of thought reasoning
Researchers attempt to elicit chain-of-thought (COT) reasoning from AI taggers to improve consistency with human preferences.
Researchers replace the standard ending prompts (e.g. replace “Preferred Summary=” with “Consider the coherence, accuracy, coverage, and over-all quality of each summary and explain which one is better. Rationale:”), and then decode an LLM reply.
Finally, the researchers concatenate the original prompt, the response, and the original ending string “Preferred Summary=” and follow the scoring procedure in Section 3.1 to obtain the preference distribution. See the figure below for the specific process.

In zero-shot hints, the LLM gives no examples of what the inference should look like, while in the few-shot hints, the researchers provide examples of COT inferences for the model to follow. See the figure below for an example.

Self-Consistency (Self-Consistency)
For thought-chain cues, the researchers also experimented with self-consistency—a technique that improves thought-chain reasoning by sampling multiple reasoning paths and aggregating the final answers produced at the end of each path.
We sample multiple thought chain rationales using a non-zero decoding temperature, and then follow the method in the previous section to obtain the LLM preference distribution for each thought chain. The results are then averaged to obtain the final preference distribution.
Reinforcement Learning for AI Feedback
After the preferences are labeled by the LLM, a reward model (RM) is trained to predict the preferences. Since the researchers’ method produces soft labels, they use the cross-entropy loss of the softmax of the RM-generated reward score instead of the loss mentioned in the reward model.
Softmax converts RM’s unbounded scores into a probability distribution.
Training RMs on AI-labeled datasets can be seen as a form of model distillation, especially since researchers’ AI taggers are often larger and more powerful than RMs.
Another approach is to bypass the RM and directly use the AI feedback as the reward signal in RL, although this approach is more computationally expensive since the AI tagger is larger than the RM.
With the trained RM, the researchers performed reinforcement learning using a modified version of the Advantage Actor Critic (A2C) algorithm adapted to the field of language modeling.
evaluate
The researchers evaluated their results by three metrics – AI tagger alignment, pairing accuracy and win rate.
AI tagger alignment is used to measure the accuracy of AI tagging preferences relative to human preferences.
For a single example, convert soft AI labeled preferences into a binary representation. Assign 1 if the annotation is consistent with the target human preference, and assign 0 otherwise.
Pairwise accuracy is a measure of how accurate a trained reward model is relative to a held-out set of human preferences.
Given a shared context and a pair of candidate responses, the pairing accuracy is 1 if RM scores the preferred candidate higher than the non-preferred candidate based on human annotations. Otherwise the value is 0. This quantity is averaged over multiple examples to measure the overall accuracy of the RM.
Win ratio evaluates the end-to-end quality of two strategies by measuring how often humans prefer one strategy to the other.
Given an input and two generation results, the human annotator chooses which generation to prefer. The percentage of instances where strategy A outperforms strategy B is called the “A vs. B win rate”.
Experiment Details
The researchers used a filtered Reddit TL;DR dataset curated by OpenAI.TL;DR Contains about 3 million posts from Reddit on various topics (also known as “subreddits”) as well as summaries of posts written by the original authors.
The data is also filtered by OpenAI to ensure high quality, which includes using a whitelist of Reddit topics that the general public can understand.
Also, only posts with 24 to 48 callouts in the feed are included. The filtered dataset contains 123,169 posts, about 5% of which serve as validation set.
More details on the dataset can be found in the original paper. Additionally, OpenAI curated a human preference dataset from the filtered TL;DR dataset.
For a given post, two candidate summaries are generated according to different strategies, and the taggers are asked to score their favorite summaries. The total dataset contains approximately 92k pairwise comparisons.
LLM callout
To evaluate the effectiveness (e.g., hints, self-consistency) of AI annotation techniques, researchers select examples from the TL;DR preference dataset, where human annotators would prefer abstractions with higher confidence.
The researchers evaluated the AI tagger alignment on a random 15% subset of the training split of the dataset for faster experiment iterations, generating 2851 evaluation examples.
For reward model training, TL;DR the full training split of the preference dataset is annotated by LLM and used for training regardless of the confidence score.
model training
The researchers trained the SFT model on the OpenAI filtered TL;DR dataset using PaLM 2 Extra-Small (XS) as an initial checkpoint.
Then, the researchers initialize RMs from the SFT model and train them on OpenAI’s TL;DR human preference dataset.
For the results in Tables 1 and 5.1, the researchers used PaLM 2L to generate AI-annotated preferences, using “OpenAI + COT 0-shot” hints (, no self-consistency, and then trained the RM dataset on the full preferences.
For reinforcement learning, researchers use Advantage Actor Critic (A2C) to train policies. Both policy and value models are initialized from the SFT model. The researchers use the filtered Reddit TL;DR dataset as an initial state to roll out their strategy.
human evaluation
The researchers collected 1200 human ratings to evaluate RLHF and RLAIF strategies. For each rating task, evaluators receive a post and 4 summaries generated from different strategies (one each for RLAIF, RLHF, SFT, and human reference) and are asked to rank them in order of quality, without any linkage .
Posts are taken from the holdout set of the TL;DR supervised fine-tuning dataset, which was not used for any other evaluations. Once these rankings are collected, the odds of any two strategies can be calculated.
50% win rate, tie
RLAIF vs. RLHF
At the beginning of the article, Google has introduced the advantages of comparing RLAIF with RLHF, and the results show that the two methods have similar performance.
Specifically, human raters preferred RLAIF to baseline SFT 71% of the time.RLHF outperformed SFT 73% of the time.
The researchers also directly compared the win rates of RLAIF and RLHF and found that they were equally popular — that is, both had a 50% win rate.
To further understand the differences between the two strategies, Google performed a qualitative comparison of the summaries it generates.

Additionally, they compared RLAIF and RLHF summaries with human-written reference summaries.RLAIF produced summaries better than the reference summaries 79% of the time, and RLHF results outperformed the reference summaries 80% of the time.
It can be seen that the difference in winning rate between RLAIF and RLHF and the reference abstract is only 1%, and there is no significant difference.
It is worth noting that the researchers also found that the frequency of hallucinations in the RLHF strategy tends to be higher than that in the RLAIF, as shown in the text marked in red in the table above.
After controlling for summary length, the RLAIF and RLHF strategies still outperform the baseline SFT and achieve similar win rates.

These results demonstrate that RLAIF does not need to rely on human annotation and is a viable alternative to RLHF.
tips
In the use of hinting techniques, the Google team tried three types of hinting techniques, preamble specificity, CoT, and few-sample context learning.
It was found that with hints from detailed OpenAI preambles and CoT inference, AI taggers can achieve 78% agreement.
Whereas contextual learning does not improve accuracy and may even make it worse.

Self-consistency
The researchers performed self-consistency experiments using 4 and 16 samples, with a decoding temperature of 1.
Sampling multiple thought chain principles with T = 1 results in low agreement with human preferences.

The size of the large model tagger
The study also found that scaling up the parameter scale of large-model taggers may result in higher-quality preference annotations.
Number of preferred examples
How does the accuracy of the reward model vary with training examples?
The researchers found that after training on thousands of examples, the performance of the reward model was close to that of training on the full dataset.

in conclusion
The researchers demonstrate that RLAIF can produce comparable improvements to RLHF without relying on human annotators.
While this work highlights the potential of RLAIF, there are still some limitations.
First, this study only explored the summarization task, and further research is needed on the generalization to other tasks.
Second, the researchers did not estimate whether LLM inference has an economic cost advantage over human annotation.
In addition, there are some interesting questions to investigate, such as whether RLHF combined with RLAIF can outperform a single method, how well using LLM to directly assign rewards, whether improving AI tagger alignment will translate into improved final policies, and whether the policy can be further improved using an LLM tagger of the same size as the policy model (ie, whether the model can “self-improve”).
Hot discussion among netizens
Google published two papers on RL:
1. RLAIF: Training Reward Models Similar to Human Feedback
2. ReST: Facilitating Self-Training Using Generative Models Combining these two papers can satisfy data-hungry AI algorithms

Half a month ago, Google DeepMind just proposed a new algorithm ReST, in order to make the large-scale language model consistent with human preferences.
Specifically, through offline reinforcement learning methods, the translation quality of large language models is improved to better meet human preferences.

Based on qualitative tests, Anthropic’s Claude model appears to be weaker than GPT-4, according to one researcher. This could be caused by RLHF/RLAIF methods or pre-training. It is unclear whether these methods generalize better in practice, even if they perform better on academic benchmarks.

I wouldn’t say this reduces the importance of human annotation, but one thing is certain, RL fed by AI can reduce costs. Human annotation is still extremely important for generalization, and the RLHF+RLAIF hybrid method outperforms any single method.

Most netizens think that the paper is a great breakthrough, but some netizens think that there seems to be no essential difference between this and the RLAIF in Constitute Claude proposed by Anthropic a few months ago.

References: