Trends in Large Language Models in the Age of Generative AI

1. Introduction

Since the advent of ChatGPT^[1], which is based on generative AI technology, generative AI is rapidly spreading across all industries, including the economy and society. Generative AI is an artificial intelligence technology that actively generates results based on user demand. It helps individuals and businesses be more productive, can be used for professional-level creative endeavors, and makes artificial general intelligence (AGI)^[2] more attainable. LG AI Research is conducting various methodologies based on Large Language Models(LLMs) and research to apply them to actual industrial sites.

We will take a closer look at one of the signature technologies of generative AI, LLMs, and consider how we can gain a competitive edge in the age of generative AI.

2. Data

2.1 Training Data

Training data plays an essential role in understanding and processing various phenomena in language. As the size of language models has continued to increase in recent years, larger amounts of data are being used to train large language models. For example, training GPT-3^[3], a model with 175 billion parameters, took about 300 billion tokens. However, the recently released LLaMA-2^[4] model with 70 billion parameters used 2 trillion tokens to train, so there is a competitive trend to increase not only the size of the model but also the size of the data used for training. Many companies are participating in large-scale language model training, and are opening up various data sources.

1. CommonCrawl: CommonCrawl^[5] is a non-profit organization that holds and discloses web crawling data from 2008 to the present. C4^[6], Redpajama^[7], RefinedWeb^[8], etc. are data that have been preprocessed differently using portions of data from CommonCrawl sources and are used by many large language models in training. Due to the nature of web data, a variety of topics, vocabulary, and writing styles are contained within.

2. Wikipedia: Wikipedia^[9] is online encyclopedia data. It consists of a variety of topics and is categorized as high-quality data because of its refined content.

3. Code: Code is divided into program code and program code-driven question-and-answer data. In the former case, there is TheStack^[10] data sourced from Github, a code repository, and in the latter case, there is StackExchange^[11], where code-related questions and answers are actively held.

4. Other: Data such as Bookscorpus^[12], which consists of books, Arxiv^[13], which covers academic content, PubMedCentral^[14], and USPTO^[15], are also used for training.

2.2 Evaluation Data

The evaluation of large language models is multi-faceted and uses many different kinds of tasks and evaluation metrics. Evaluation standards can be set in various ways, but in this paper, they are categorized into language generation ability, knowledge utilization, and reasoning ability and introduce representative evaluation data^[16].

1. Language generation: There is LAMBADA^[17], a task that completes sentences, and XSum^[18], which is related to summarization.

2. Knowledge utilization: There are two types of QA tasks: closed-book QA tasks, which aim to assess the ability to answer questions based on common sense or knowledge, and open-book QA tasks, which require answering questions using only previously learned knowledge. Representative data include Natural Questions^[19], ARC^[20], TruthfulQA^[21], etc., and for open-book QA, external sources like Wikipedia are used to evaluate models.

3. • Reasoning: This ability consists of tasks that assess whether the model can make sense of the information it is given or use the knowledge it has learned to make a series of logical inferences to achieve a suitable result. Examples include PIQA^[22], HellaSwag^[23], WinoGrande^[24] for general common sense reasoning, and GSM8K^[25] for testing mathematical logic.

In addition, MMLU^[26] and BIG-bench^[27]data, which evaluate knowledge in various domains, are widely used, and there is also data that three-dimensionally evaluate the accuracy, robustness, equality, and efficiency^[28]. When the purpose calls for a specific domain-oriented model evaluation, such as code or bio, model performance is also evaluated with an appropriate dataset.

2.3 Training Data Processing and Refinement

After collecting data, the data goes through a process of refining it into a form suitable for training a language model. Along with the quantity of data, the quality of the data also affects the performance of the language model, so a variety of preprocessing methods have been proposed^{[3, 8, 29]}. The data processing process largely consists of filtering, removal of duplicates, and de-identification steps to improve text quality.

1. Filtering: Filtering can be divided into rule-based and model-based filtering. For example, web crawl data contains excessive repetition of words and strings, repetition of certain phrases, and unnecessary phrases that indicate web formatting, which can be handled by rule-based filtering. On the other hand, model-based filtering is a method of classifying quality and filtering out low-quality documents.

2. Removal of duplicates: Among collected documents, there may be documents with duplicate content, and this duplication can lead to poor performance of the language model and inefficient training^[30]. To remove duplicates from documents, duplicates are typically removed by determining if the content of the documents is an exact match or similar, and if additional information is available, such as a URL, duplicates are removed by verifying that the URL matches.

3. De-identification: Vast amounts of web data can contain sensitive information (e.g. social security numbers, email addresses, etc.). To prevent such leakage of personal information, the relevant part is removed or a de-identification process is started.

3. Pre-Trained Language Model

Pre-trained language models are divided into encoders, decoders, and encoder-decoder types depending on the model structure. In this article, the focus will be on the decoder model and the encoder-decoder model, which have received a lot of attention recently.

3.1 Decoder Pre-trained Model

The decoder model is a unidirectional transformer language model that utilizes only the decoder from the Transformer^[31], with the GPT model being a representative example^[32]. Since it trains by predicting the next token from a given set of tokens, it can be pre-trained on unlabeled corpora, and it performs well on the generation task of predicting the next word after a given text.

As the GPT model gradually developed, the number of model parameters and the amount of learning corpus increased significantly, and GPT-3, a model with a massive 175 billion parameters, is pre-trained with 300 billion large tokens, introducing the concept of a large language model^[3].

Before the advent of large models, natural language processing problems were solved by training a model from scratch with training data for each task or by further fine-tuning a pre-trained model. However, large language models have the disadvantage that additional training costs are very expensive due to the large model parameters. At this, GPT-3 has shown that in-context learning can solve many problems without additional training. In-context learning is a method in which you input a natural language prompt consisting of a task description, examples consisting of task questions and answer pairs, and questions, and the model generates an answer based on the given input. GPT-3 achieved performance that surpassed existing fine-tuned models through in-context learning in several tasks^[3].

Afterward, to further improve the response quality of large language models, instruction tuning^[33] and reinforcement learning with human feedback^[34] were proposed. Instruction tuning showed that fine-tuning a language model for various tasks with questions in the form of instructions as input and the answers to be generated as output improved model performance, even for unseen tasks. Reinforcement learning with human feedback showed that by rewarding answers to input questions that are more likely to be preferred by humans, the model generates answers that are more likely to be preferred by humans.

Overseas, OpenAI, Google, Meta AI, and others have announced GPT-4^[35], PaLM-2^[36], and LLaMA-2^[4], and utilized them to provide services such as ChatGPT and Bard. In Korea, models such as LG AI Research’s EXAONE, NAVER’s HyperCLOVA, SK Telecom’s A., NCSoft’s VARCO, Konan Technology’s Konan, Kakao’s KoGPT, and KT’s Mi:dm have been announced. Among these, EXAONE is a bilingual model trained on a large Korean and English corpus, which shows high performance in both Korean and English, the two main languages used by domestic professionals. It is utilized in the EXAONE 2.0 Universe service, a conversational AI platform for professionals that generates accurate, evidence-based answers to questions in areas requiring expertise.

3.2 Encoder-Decoder Pre-Trained Model

The encoder-decoder model is a model that uses both the two proposed structures, encoder and decoder, in the transformer. Unlike encoder-only models such as BERT^[37], encoder-decoder models are characterized by the fact that the input and output parts have different structures, allowing heterogeneous data with different shapes to be used as input and output. The encoder-decoder model utilizes a cross-attention structure to connect input information to output. Technically, it utilizes an encoder to convert a given input into a vector representation called a hidden state, and exchanges information through cross-attention between the converted input vector and the decoder structure to generate the final output.

T5^[38], a representative of the encoder-decoder model, utilizes a span corruption objective to perform dictionary training on a large number of web documents. The span corruption objective is a way to mask regions across multiple tokens when training a language model, rather than masking for a single token. The span corruption objective is a technique that has shown good performance by increasing the difficulty of the problem the model has to learn.

A subsequent study reported that the structure of the T5 model can be utilized to process other forms of non-linguistic input, such as speech^[39]. In the image domain, the VIT^[40] model has achieved high performance by learning images with an encoder-decoder structure, and the encoder-decoder structures are rapidly replacing traditional CNN structures.

As a pre-trained language model has a high dependency on the language it is trained on, a pre-trained model trained on a Korean corpus is required to perform the Korean task. However, there are only a few encoder-decoder models available for Korean language tasks, especially for large language models with hundreds of millions of parameters. In addition, performance issues have led researchers to choose Google’s multilingual T5^[41] model over the publicly available Korean model.

LG AI Research researched and developed its own encoder-decoder model and achieved results that exceeded the Google T5 model in various tasks in both Korean and English. In the comparison between models with multi-task tuning techniques, significantly higher performance was achieved compared to Google FLAN-T5. Additionally, the performance of the EXAONE model was recognized by achieving first place on the KorQuAD 1.0 leaderboard^[42].

4. Instruction Tuning

4.1 Instruction Tuning for LLM

Along with the evolution of large language models comes the development of zero-shot abilities to perform new tasks not learned by the model. These characteristics suggest that large models are inherently multi-task learning with the training goal of predicting the next token during pre-training, which allows it to acquire the ability to perform new tasks^[43].

Recent research has proposed instructional tuning methods that explicitly learn different tasks as a methodology to improve zero-shot performance^{[33, 44]}. This method provides more meaningful guidance to the language model by explicitly learning the questions and answers for each task. Instruction tuning has two main characteristics: it learns multiple tasks, and it learns by adding instructions that contain descriptions of the tasks. Because of this, instruction tuning is also known as multi-task prompted training^[44].

4.2 Task Clusters

In instruction tuning, NLP tasks are composed of many different datasets, and when these datasets are grouped together based on a common task type, they are called task clusters. The following is an example of a 12-task cluster defined by FLAN^[33].

[Figure 1] FLAN’s task cluster classification^[33]

 

To evaluate the effectiveness of instruction tuning, the model is trained excluding certain task clusters, and then the performance of the dataset on those task clusters is measured. The more diverse these task clusters are, the higher the performance of the model.

 

4.3 Instruction

Instructions, also called templates and prompts, are natural language descriptions of the characteristics of a given task. This trains the language model what answers to generate for that task, and by learning different instructions, it is able to understand and generate answers when given new instructions from the user. Each dataset contains multiple instructions, and the more diverse the instructions, the more robust the model.

 

4.4 Cross-lingual Instruction Tuning

The most challenging part of instruction tuning is that it requires a rich dataset to train on, which in turn requires a wide variety of instructions. For English, various instruction-tuning datasets^{[33, 44, 45]} have been proposed, but for most languages other than English, such datasets are often difficult to obtain. To address these issues, cross-lingual instruction tuning has been proposed, in which instruction tuning is performed on a rich language (e.g., English) in a multilingual model, and then the effect is applied to the other language (e.g., Korean)^[46]. This can alleviate the difficulty of building an instruction-tuning dataset by exploiting cross-language knowledge transfer. LG AI Research developed this research to solve the problem of scarce data in Korean and confirmed improvements in performance by efficiently constructing instruction tuning data.

 

5. Reinforcement learning from human feedback

Large pre-trained language models based on decoder structures, such as GPT-3, have shown remarkable performance on a number of natural language processing problems, but they often produce harmful answers or exhibit hallucinations that contradict user expectations^{[47, 48, 49]}. This is mostly interpreted as being due to the fact that the objective function used to train the generative model (either the next-word prediction objective or the span corruption objective) does not fully match the satisfaction of real users^{[50, 51]}.

Methodologies for injecting user feedback into generative models have been actively researched to bridge this gap and provide a better user experience^{[51, 52, 53, 54, 55]}, and in particular, the commercial success of ChatGPT has highlighted the importance of human preference alignment. This paper focuses on reinforcement learning techniques using Proximal Policy Optimization (PPO) that have been used to inject user feedback into InstructGPT^[51] and the LLaMA-2-chat model^[4].

 

5.1 Human preference data

Data for training the model is largely secured based on harmlessness and helpfulness. In a model where existing instruction fine-tuning has been completed after the model generates a range of multi-turn outputs for random inputs similar to real-world user inputs, crowd workers rank the different outputs. Here, as the crowd workers only need to rank the multiple outputs, a large amount of user satisfaction data can be obtained much more efficiently than generating instruction fine-tuning data for existing supervised learning.

Data in the form of rankings reflecting user satisfaction is transformed into a distribution of user preferences through the Bradley-Terry model^[56], which expresses the priority of user satisfaction in two sentences. Additionally, data in the form of rankings for n outputs can be utilized as _n C_2 user preference data. It is known that large amounts of user preference data are required to reliably inject user feedback into large-scale language models such as the LLaMA-2-70B, and in the case of the LLaMA-2, about 1.42 million user preference data were utilized^[55].

 

5.2 Reward model

With the large amount of user preference data constructed through the above process, it is possible to train a reward model that expresses user preferences as scalar values between 0 and 1 for input sentences. For large-scale language models, the model is trained with instruction fine-tuning completed as the starting weights, and the objective function of the reward model is constructed based on the difference between the reward values of the more preferred input sentences and the reward values of the less preferred sentences.

Reward models are also known to be parameter scalable^[57]. InstructGPT used a language model with 6 billion parameters for its reward model, and LLaMA-2 used a language model with 70 billion parameters. Furthermore, LLaMA-2 was expected to have the effect of reducing the trade-off phenomenon between “harmlessness” and “usefulness” by constructing a reward model that expressed “harmlessness” and a reward model that expresses “usefulness.”

 

5.3 Proximal Policy Optimization

Since it is very difficult to train a generative AI model by directly differentiating the user preferences represented by a well-trained reward model, reinforcement learning techniques are borrowed to inject user preferences into the generative AI model. In general, reinforcement learning techniques have unstable learning curves, especially for natural language processing models^[54]. Methodologies designed to mitigate the instability problem and make learning more stable are the Trust Region Method^[58] and the Proximal Policy Optimization (PPO)^[59].

PPO is a methodology that trains a policy model in the direction of the objective function of reinforcement learning but constrains it with KL-divergence to prevent the distributions of the trained model and the original model from being too different. The PPO methodology applied to InstructGPT or LLaMA-2 sets a constraint that as the policy model continues to be trained, the KL-divergence from the distribution of the instruction-fine-tuned model, which was the initial starting point of the model, does not deviate beyond a certain level.

 

5.4 Examples and Evaluations of Reflection of User Feedback based on Reinforcement Learning

The methodology of incorporating user feedback using PPO reinforcement learning techniques has been applied to OpenAI’s ChatGPT, Anthropic’s Claude 2, and MetaAI’s LLaMA-2-chat, among others, demonstrating the commercial value of user preference-infused models. ChatGPT stated it is iterating and deploying the process of injecting the user preferences mentioned above to continuously improve usability.

User preference-infused models can be evaluated with benchmark datasets that evaluate existing instruction fine-tuning models. In particular, the “Open LLM Leaderboard”^[60], which collects and evaluates four benchmarks (ARC-challenge, TrustfulQA, MMLU, and HellaSwag), and MT-bench^[61], which generates multi-turn conversations and then evaluates them using GPT-4, are frequently used. To evaluate user preferences more directly, the "Chatbot arena" approach is also used, which evaluates preferences among language models using crowd workers and ELO ratings^[61].

 

6. Model Lightweighting

Attempts to lightweight already-trained neural network models without compromising performance have been well studied in the past. In particular, the latest language models are very large, which makes them very expensive to run in actual service, so lightweighting them is very important. This section describes quantization, knowledge distillation, pruning methods, and recent research trends.

 

6.1 Quantization

Quantization reduces the numerical precision of parameters while maintaining the performance of the neural network model. Typical neural network models use 32-bit or 16-bit floating points, which can be reduced to 8/4/2-bit while maintaining performance. Commonly used integer quantization is as follows.

$$\begin{array}{c}{x}_q=\text{quantize}(x,b,s,z)=\mathrm{clip}(\mathrm{round}(s\cdot x+z),-2^{b-1},2^{b-1}-1)\\ \hat{x}=\mathrm{dequantize}(x_q,s,z)=\frac{1}{s}(x_q-z)\end{array}$$

This means that by taking the maximum and minimum values of a parameter, multiplying the real number by the scale value, adding the zero value, and quantizing to get an integer value, this will be able to replace the original real value parameter. In actual reasoning, the integer, scale, and zero values are stored and restored to their original real values for use.

 

6.2 Knowledge Distillation

Knowledge distillation is a method of achieving similar performance between two models when there is a teacher model to pass on knowledge from, and a student model to receive the knowledge. When transferring knowledge, training to make two models resemble each other by passing not only token information in the output but also additional information, such as the probability of occurrence, can yield higher performance than training the model to receive the knowledge directly.

 

6.3 Pruning

It is known that picking out the less important parameters of a neural network model and excluding them will preserve the model’s performance. In some cases, performance can be maintained by excluding more than half of the parameters, in which case only a subset of the parameters need to be stored and calculated.

 

1. Unstructured pruning: This is a method of excluding parameters of low importance regardless of the model structure. However, there is no standard way to represent the model, so there is no memory and speed advantage without special hardware support, only a parameter storage advantage.

2. Structured pruning: This is a method of excluding parameters while changing the structure of the model. There are memory and speed benefits to reducing the size of the model, but has the disadvantage of significantly decreasing performance.

 

7. Linkage with Information Retrieval Technology

Innovation in generative large language models is occurring rapidly with many of the aforementioned technologies. However, the possibility of reflecting real-time information or hallucinations that plausibly create untrue content cannot be ruled out. Figure 2 shows the results of running ChatGPT, which shows that ChatGPT has only learned information up to September 2021 and cannot answer questions after that date. What’s more, in Figure 3, we also see the phenomenon of giving answers that are not true as if they were.

 

[Figure 2] Limited information learning according to learning time

[Figure 3] Limited information learning according to learning time

 

To compensate for these shortcomings, many recent attempts have been made to address the problem by linking information retrieval technologies that reflect real-time information or provide rationales for answers to large language models. Retrieval Augmented Generation (RAG)^[62] consists of two modules: a retriever and a generator. The retriever will find the most relevant documents for a given query, and the generator will generate answers from a large language model based on the documents found by the retriever.

 

[Figure 4] RAG structure ^[63]

 

LG AI Research is conducting research on RAG technology to compensate for the shortcomings of large language models. The QASA (Advanced Question Answering on Scientific Articles) model and dataset were released as a result of research on answering questions in scientific articles such as papers^[64]. QASA goes through a three-stage process of associative selection, rationale generation, and systematic composition, as shown in Figure 5 below. The associative selection stage selects passages that are relevant to the question from the candidate documents found through information retrieval, and the rationale generation stage generates rationales for answering the question based on the content of each passage, and the final stage, systematic composition, combines multiple rationales to finally generate an integrated response.

When QASA trained the publicly available T5 (3 billion parameters) model on the dataset used in the experiment, it achieved better performance than InstructGPT (175 billion parameters). This study has the advantage of providing a detailed, passage-level rationale that serves as the basis for answers, minimizing any possible hallucinations, and providing rationale so that users can form a second opinion. Based on this research, LG AI Research is applying its own giant language model, EXAONE, to various question-answering services based on specialized literature such as internal security documents, chemical/bio papers, patents, and legal documents.

 

[Figure 5] Overview of QASA^[64]

 

8. Conclusion

We introduced the technology trends of large language models, the signature technology of generative AI. Large language models are created by leveraging large amounts of textual data to acquire the basic knowledge to understand a language through a pre-training process, followed by instructional tuning to help users understand instructions and perform tasks, and user feedback based on reinforcement learning to generate human-preferred answers. Large language models can be utilized in a variety of fields, including healthcare, law, education, marketing, finance, etc., and with continued technological developments, they are expected to revolutionize and advance many more fields in the future.