[NeurIPS 2023] Latest LLM Research Trends

At NeurIPS 2023, many sessions were held in the field of large language models (LLM). In addition to research on improving LLMs' performance, various topics were covered, including reliable evaluation methods, alignment between LLMs and humans, optimization in training and inference, various applications using LLMs, and privacy/bias that can be caused by LLMs. This post will look at the latest LLM research trends presented at NeurIPS 2023 and provide an in-depth look at some of the most impressive research on LLM pre-training data.

Section 1. Trends in LLM Research from NeurIPS 2023 Oral Papers

Oral and poster presentations were held over three days during NeurIPS 2023. Seventy-seven oral papers were presented on the following schedule (local time).

2023.12.12 Morning: Reinforcement Learning(1), Datasets & Benchmarks(1), Tractable Models Afternoon: Efficient Learning Objects, Neuroscience, Vision Causality Privacy 2023.12.13 Morning: Neuro NLP/Tools, Diffusion Models Afternoon: Optimization Datasets & Benchmarks(2), CoT/Reasoning 2023.12.14 Morning: Privacy/Fairness, Probability/Sampling, Vision(1) Afternoon: LLMs, Reinforcement Learning(2), Vision(2), Theory(2)

It is interesting to note that, due to the keen interest in LLMs, a lot of LLM-related research was featured on topics other than LLMs on December 14. This section will present some of the most notable LLM-related papers presented orally.

1. LLMs

1. Are Emergent Abilities of Large Language Models a Mirage?^[1]

The author argues that the emergent abilities observed as LLMs increase in size should not be interpreted as any feature that actually applies to a specific model or task but rather as a phenomenon that shows rapid qualitative growth on a particular task due to the metric used to evaluate model performance. They emphasize the need to be careful in the construction of benchmarks and metrics, as the characteristics may vary depending on the number of samples used in the evaluation and the characteristics of the metrics used by the researchers (nonlinearity, discontinuity).

1. Jailbroken: How does LLM Safety Training Fail?^[2]

Nowadays, LLMs are trained in a helpful and harmless manner. Despite this, in real-world situations, there are cases in which LLM-generated results may not lead to a safe answer based on the input of a prompt with specific malicious intent. The author of the paper explains why this behavior is observed despite safety-conscious learning with two hypotheses. 1) When there is a conflict between pre-training and instruction-following, or 2) when the input is out-of-distribution data in the safety training process, but is data that could have been included in the pre-training phase.
This suggests that safety training is still weak and that simply increasing data and model size is not the correct way to learn.

2. NLP/Tools

1. Toolformer: Language Models can Teach Themselves to Use Tools^[3]

LLMs show outstanding performance in a wide range of fields but often perform poorly in math and fact-based information reasoning. Therefore, the author of this paper proposes to create demonstration datasets that can be learned with self-supervised learning to help models imagine tasks that can be done much better with external API tools and to learn to make good use of these tools. One of the follow-ups of Toolformer is ToolkenGPT, which is explained below.

1. ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings^[4]

Utilizing fine-tuning or in-context learning to allow LLMs to use various external tools introduces disadvantages, such as adaptation to new tools or limitations on the length of demonstrations. To compensate for this, the author introduces Toolken, a tokenization of tools. They propose a framework that learns only Toolken embeddings, appends them to the language model head, and executes the corresponding tool to generate an answer when Toolken is encountered in the next token prediction.

3. Efficient Learning

1. QLoRA: Efficient Finetuning of Quantized LLMs^[5]

This study is one of the methodologies of parameter efficient fine-tuning (PEFT) to fine-tune some model parameters for efficient fine-tuning of LLMs. The authors propose a methodology that uses the 4-bit NormalFloat data type, double quantization, and a paged optimizer to fine-tune the trained model down to 4 bits.

1. Scaling Data-Constrained Language Models^[6]

Unlike other deep learning trainings that use multi-epoch, LLM training has been using a single epoch. However, there are times when the data available for training is very limited, such as for low-resource languages, and data must be learned repeatedly to fill the insufficient learning. This paper analyzes the impact of learning LLMs with multi-epoch in such data-constrained situations and proposes a scaling law.

4. Datasets & Benchmarks

1. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models^[7]

As LLMs are used for various purposes, evaluating aspects beyond linguistic and knowledge competence, such as model robustness and calibration, in addition to benchmarks that assess a model’s generalized language ability or high-difficulty tasks, is becoming increasingly important. To evaluate the trustworthiness of the model, this paper constructed a comprehensive evaluation benchmark for toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness. Furthermore, when evaluating GPT-4 and GPT-3.5, it points out that despite GPT-4’s better overall evaluation, its ability to follow instructions makes it more likely to be compromised when jailbreaking.

5. Reinforcement Learning

1. Direct Preference Optimization: Your Language Model is Secretly a Reward Model^[8]

In traditional deep learning, only the benchmark performance on specific tasks is considered when measuring a model’s performance. In contrast, the performance of the LLMs of 2023 is not only about general benchmark performance but also about the ability to follow instructions with a high match to human preference. Training reward and policy models are the main methods used to increase this alignment with human preference. This paper proposes a much more stable and computationally lighter method by optimizing the policy directly via binary cross-entropy.

Section 2. Research on LLM Pre-training Data

As seen above, there are many important research directions for LLM learning. However, there is one area in which there is no clear rule of thumb yet and which is hidden in many technical reports: pre-training data.

LLMs are no exception to the fact that high-quality data greatly determines model performance. Therefore, a series of steps are taken to organize the data helpful for LLM learning. The final pre-training data is constructed by filtering, de-duplicating, and de-identifying documents in the data and weighting them according to the training data source.

This section will cover the research presented at NeurIPS 2023 that impressed us the most regarding LLM pre-training data.

1. [DoReMi:Domain Reweighting with Minimax Optimization]^[9]

1. Motivation

As written in a previous LG AI Research blog post, “Generative AI Era: Technology Direction of Large Language Models (LLMs),” there are various source data that can be used as training data, including web data CommonCrawl, Wikipedia, code source data, patents, and data with academic content. Let’s look at the data organization table from the well-known The Pile dataset paper^[10] in Figure 1 below.

Figure 1. Data organization table in ThePile dataset paper^[10]

To make sense of the table, let’s explain each term. A component is a variety of data sources; the effective size is the raw size multiplied by epochs, and the weight is the effective size for each domain divided by the total effective size. For this dataset, rather than calculating the domain weight first, the number of epochs is set to include the duplicates in the final training data for data that can be considered high-quality, calculate the size of each dataset, and then calculate the domain weight.

For example, Wikipedia can be included three times in the final pre-training data. However, as seen in Figure 2, it is highly heuristic to determine which data is of which quality for different data. In addition to The Pile, papers such as PaLM^[11] and GLaM^[12] tune weights based on downstream task performance, but this requires multiple experiments, which require a lot of computing resources. There is also the problem of overfitting the selected downstream task.

2. Method

Rather than relying on a downstream task, this paper proposes DoReMi (Domain Reweighting with Minimax Optimization), a method for algorithmically calculating domain weights in such a way that the model learns to perform well on each data source it uses to learn. Before we get into the full methodology, here are the important points.

1. Point 1: Use the domain weight from the smaller model

DoReMi first obtains the domain weight by optimizing the loss of small reference and proxy models, then learns a much larger model with the data using the domain weight.

1. Point 2: Learn to do well across all data sources

The somewhat abstract “learn to do well across all data sources” leads to group distributionally robust optimization (group DRO), which updates in a way that reduces the worst-case loss in each domain. The idea of group DRO is that for datasets that can be divided into groups, instead of updating with a loss averaged over the data used for training, we optimize with the loss for each group. This is because when using average loss, the loss of a specific group may be small, but a specific group may have a large loss.

Coming back to the DoReMi framework, the goal is to get the domain weight. To do this, there are three steps.

1. Step 1: Learn a small reference model to base losses on. The data you use to train the initial reference model can use uniform weights.
2. Step 2: Learn the proxy model as a group DRO to get the domain weight.

 

Figure 2. DoRemi Domain Reweighting

 

The difference between the loss of the reference model trained with an arbitrary domain weight and the loss of the proxy model, which is updated step-by-step, is used to optimize the domain weight and proxy model in order, and the final domain weight is obtained. The domain weight is updated to increase for domains with a significant difference in loss, and the proxy model is updated to reduce the worst-case loss difference multiplied by the updated domain weight. Worst-case loss is defined as shown in Figure 4 below.

 

Figure 3. Defined worst-case loss

 

1. Step 3: The data is then sampled with the obtained domain weight to train the actual target model.

 

3. Evaluation

The size of the reference and proxy models used in the experiment is 280M, and the target model is 8B. The Pile's baseline is The Pile's domain weight, and GLaM's baseline is the uniform weight given that it is tuned for the downstream task. Below is the 0-shot average accuracy (exact match) for the five benchmarks used in GPT-3 (TriviaQA, Natural Questions, WebQuestions, SQuAD v2, and LAMBADA). We can see that The Pile scores above Baseline after 1 DoReMi, GLaM scores above Baseline after 2 DoReMi, and performs on par with GLaM's Weight, which can be seen as Oracle.

 

Figure 4. Comparing the performance of The Pile and GLaM Dataset 

 

Domain weight obtained through DoReMi tends to have a somewhat different value from the initial value of The Pile.

 

Figure 5. Comparing domain weight changes after DoReMi

 

4. Discussion

While we briefly demonstrated the size-dependent ablation of the reference model, an in-depth analysis of how the reference model learns and its size is expected to be a noteworthy research topic. The study also suggested that the degree of granularity of the data source domain can make a difference in performance. It is an impressive study that makes one rethink the domain weight area, which, despite being an important area in the field, has been relegated to heuristic numbers and short descriptions in many studies.

Section 1 of this post explored the latest research trends in LLM through oral papers at NeurIPS 2023. Section 2 looked in-depth into a somewhat obscure study that proposed a methodology to find the domain weight of pre-training data algorithmically.

At NeurIPS 2023, we were able to watch tutorials, panel talks, and research presentations related to LLMs, which gave us a first-hand look at the keen interest in LLM from researchers around the world. As we look forward to LLMs’ tremendous technological expansion potential, LG AI Research will continue to develop learning and data organization methodologies, efficient learning and inference research, and leading research that shows high alignment with human preference to create a strong LLM foundation model.

 

▶ Want to learn more about the NeurIPS 2023 Research Blog series? (Link)