[ACL 2024] Emerging Trends and Key Insights in LLM Research

The advent of generative AI has sparked a new wave of innovation. Large Language Models (LLMs) have shown that many previously intractable problems can be solved based on knowledge and language skills learned from large amounts of natural language data.

However, as people began to actively utilize LLMs, various problems emerged. LLMs that were much more fluent often gave plausible but untrue answers, and produced ethically questionable content. Understanding these weaknesses, LLMs were maliciously attacked to produce unethical results.

Many people are contemplating ways to address the limitations of LLMs and prevent potential misuse. On one hand, there is a lot of research going on about what data to learn from, how to learn from it to develop better LLMs, and how to evaluate them. Additionally, discussions are ongoing about which capabilities should be emphasized and how LLMs should be trained to understand human intentions and deliver the desired outcomes.

At ACL 2024, tutorial sessions were held to introduce the wide range of research fields that have been expanded since the dawn of the LLM era and to explore the past, present, and future of each field. LG AI Research also participated in ACL 2024, where we presented our research achievements and engaged in discussions with researchers from around the world. This post will take a look at some of the topics related to LLM and introduce some of the impressive research. We will also showcase the research and goals of LG AI Research in this regard.

1. Automatic and Human-AI Interactive Text Generation

The first tutorial to be introduced is “Automatic and Human-AI Interactive Text Generation.” It explores the various aspects that need to be considered in order to make good use of the text generated by LLMs from a human perspective and covers how to interact with it using examples on topics such as summarizing and editing text. Among these, we will introduce the topic “How to evaluate text generated by LLMs.”

The text produced by LLMs is much more human-like and fluent than other existing language models. However, there is a need to evaluate how good and desirable the LLM’s output is. Only when we can evaluate the quality of LLM output can we make good use of the generated text.

The way we evaluate the quality of text has evolved with the advancement of Natural Language Processing (NLP) research. Traditional NLP has relied mainly on statistical methods, configuring various metrics based on N-grams, a sequence of N words, such as the BLEU score, which measures how many N-grams in the generated text are included in the correct answer, or the ROUGE score, which measures the degree of N-gram overlap between the generated text and the correct text. However, N-gram-based evaluation methods have limitations in that they are sensitive to variations in the text such as word reordering and substitutions, and do not capture sufficient semantic information.

The development of language models has led to research that seeks to evaluate text by borrowing the capacities of language models. Since the advent of BERT, metrics utilizing embedding have been proposed, most notably the BERTScore. The BERTScore is calculated by computing the cosine similarity using the BERT embedding vector of the reference sentence and the BERT embedding vector of the sentence to be evaluated. While this method of evaluation provides a better understanding of semantic information than traditional N-gram-based methodologies, it still suffers from the limitation of not fully considering context and task-specific context.

Methods for training models like RLHF for evaluation have also emerged. It has become possible to train reward models (RMs) from human feedback on which texts perform better, and these trained models can then infer what score an arbitrary text would receive from a human.

Image 1. Structure of Reward Model in Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts^[1]

This tutorial will introduce the paper “Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts”^[1]. After learning RM using people's absolute rating data, we trained the added regression and gating layers to evaluate the response from various aspects and designed it to reflect human preference well in various objectives.

"Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models"^[2], a paper published by LG AI Research, was also introduced. In the paper, the evaluation LM was configured to utilize both pairwise ranking, which compares two responses, and direct assessment, which scores a single response, to significantly increase the correlation with reference LLMs such as GPT-4 and human preference.

Image 2. Overview of Prometheus 2's evaluation methodology^[2]

As LLMs have evolved and multi-task instruction tuning has been utilized to increase the capabilities of LLMs, the trend has been to use instruction and context to solve the desired task, which is also true for LLM evaluation. Instead of measuring human preference scores as in existing RM-based methods, LLMs solve the evaluation task themselves, which is more interpretable than RM-based methodologies because LLMs can generate their own thinking and reasoning processes. This is sometimes referred to as “LLM-as-a-judge.”

MT-Bench^[3] is a typical LLM-judge methodology. After asking LLMs questions and receiving answers in two turns, the answers of the LLM to be evaluated are handed over to a judge LLM, such as GPT-4, along with the answers of a baseline LLM, to score the target LLM’s answers. In the process, MT-bench performs a pairwise comparison, single answer grading, and reference-guided grading, all of which translates into an overall score.

This LLM-as-a-judge approach has the advantage of being scalable as it is not human-evaluated, and explainable because the judge LLM explains why it made the evaluation. However, because the LLM is making the assessment, it introduces unintended biases. Existing LLMs often prefer long, detailed answers, which can lead to them not selecting simple and clear responses, even when a shorter answer would be better. Additionally, their results can change depending on where the evaluated text is placed in the prompt, and they tend to favor their own generated answers when one is provided.

While it is possible to mitigate the bias by alternating the position in the prompt or using multiple models as judges, it is not possible to completely eliminate the problems caused by the inherent bias of LLMs. This is a situation in which the way we evaluate language models is not simply solved, but is constantly evolving.

2. Vulnerabilities of Large Language Models

The second tutorial is “Vulnerabilities of Large Language Models.” As the era of the LLM dawned, so did its ubiquitous use, as well as its many vulnerabilities. There have been ongoing reports of hallucinations, or the generation of text that says things that are not true, and in some cases, harmful content.

This is due to the inherent nature of LLMs. In the process of instruction tuning the LLM’s answers in the direction that people want them to go, the LLM learns to be helpful and harmless at the same time. As these two directions are opposed to each other, it is difficult to reconcile the two objectives during the learning process.

LLMs also learn safety primarily during training but often lose this ability in data or environments that they did not learn. For example, if input is provided in base64-encoded text or a low-resource language, it will often fail to reject it properly, even if it contains harmful content. This tends to be more prevalent in larger models.

These issues can be catastrophic for systems that utilize LLMs, especially when multiple models and services are interconnected in a complex manner, as the harmful content generated by LLMs can spread throughout the entire system. Therefore, the need for fair, safe, and reliable LLMs is growing.

This tutorial introduces various techniques for attacking LLMs and explains how to defend against them. First, let's take a look at the typical LLM attack methods. Depending on the scope of the attacker’s access and the purpose of the attack, there are various techniques, which can be broadly categorized into white-box attacks, where all information in the model is accessible, and black-box attacks, where only the input and output of the model are accessible.

From the perspective of learning LLMs and deploying them as services, the main threat is black-box attacks. This is because they make LLMs give wrong answers simply by modifying the input. A typical black-box attack method is context contamination, which is the practice of embedding harmful content into the input context in advance to induce the LLM to generate harmful content based on the context. In the paper "Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations”^[4] introduced in the tutorial, it was shown that the attack success rate can be greatly increased by attacking with a technique that decorates the context as if the LLM had already answered the harmful answer.

Image 3. Examples of adversarial attacks and in-context attacks^[4]

Another method is prompt rewriting, which involves finding inputs that expose vulnerabilities in the LLM you want to attack and attacking them. PAIR^[5] attempts to input the jailbreak prompt $P$, generated by the attacker LLM, into the target LLM and then passes the generated text $R$, along with $P$, to the judge LLM to determine whether the attack $S$ was successful. If the judge LLM does not conclude that the jailbreak was successful, it provides $P$, $R$, and $S$ to the attacker LLM, which then generates a new prompt.

Image 4. PAIR attack technique ^[5]

So what can be done to defend against these attacks? Mitigation techniques can be applied at two main stages: learning and inference. Among them, it is common to apply Supervised Fine-Tuning (SFT) or Direct Preference Optimation (DPO)^[6] to learn LLMs, where data and objectives are provided to increase safety.

There is also an unlearning technique that uses Gradient Ascent (GA) to unlearn certain knowledge or instructions. While this is good at preventing the generation of harmful content because it learns by directly decreasing the probability of certain data, it has shown unstable results, as it works in the opposite direction of existing learning methods, resulting in poor results when the input is similar to the data used for unlearning. Recent studies have proposed methods that combine unlearning with additional alignment learning, such as DPO, and Safe Unlearning^[7].

Defense techniques at the inference stage often involve either preprocessing the input or postprocessing the generated text. The tutorial introduced SmoothLLM^[8], which combines both approaches. In this paper, we tried to mitigate the effects of adversarial attacks or jailbreak prompts by applying random perturbations to the prompt, each of which generates a resulting text and aggregates them.

Image 5. Comparing the inference process of SmoothLLM with that of a typical LLM^[8]

The paper presents insert perturbation, which adds a random token; swap perturbation, which replaces a token with another; and patch perturbation, which adds a token sequence. It also shows that applying only swap perturbation at a 10% level can reduce the attack success rate to less than 1%.

3. Watermarking for Large Language Model

The last tutorial is “Watermarking for Large Language Models.” A watermark is a trace that hides an imperceptible level of noise or pattern in the original data so that the data can be distinguished from it. As more and more LLM-generated data appears on the web, the likelihood of LLMs learning from their own data increases, raising concerns about model collapse. Therefore, it is desirable to verify that the text was generated by an LLM and remove it from the learning data.

So how can we distinguish between LLM-generated text? Simply put, we could train an LLM to produce specific patterns, or we could train a classifier to distinguish whether a text is LLM-generated or not. However, in the former, the quality of the text will be degraded, and in the latter, the classification performance of never-seen text will be poor due to the out-of-distribution problem. Watermarks can hide imperceptible noise in the text, preserving quality while removing the need for training and thus freeing us from the OOD problem.

Basically, watermarking text from a language model can be represented as follows. First, we create a new model $\hat{M}$ that generates watermarked text by applying a specific watermarking methodology using detection key $k$ to the model $M$. Then, a random text $y$ is checked with the detection key $k$ to see if it is a text generated by the model or not.

$\hat{M}=\text{Watermark}(M,k)$
$\begin{array}{rl}\mathrm{Detect}(k,y)& =\{\begin{array}{ll}0,& \text{if }y \text{is human-written}\\ 1,& \text{if }y \text{is generated by} \hat{M}\end{array}\end{array}$

 

Therefore, watermarking of language models is largely composed of $\text{Watermark}(·)$and $\text{Detect}(·)$.

In order for the resulting watermark algorithm to work effectively, the generated text must be of good quality, have high detection accuracy, and be robust to attacks such as text modification. The detection key $k$ must also be difficult to figure out.

So what functions should we use to get a well-working watermark structure? Kirchenbauer et al.^[9] propose a representative LLM watermarking methodology. It is also called a Green-Red watermark. The basic concept is simple. The idea is to categorize all the tokens in the LLM vocab as either green or red, and then add a small perturbation $\delta$ to the values corresponding to green tokens in the logit inferred by LLM. This results in a small increase in the probability of green tokens, and a small decrease in the probability of red tokens.

How are green tokens determined? Kirchenbauer et al. create a hash value using the N tokens before the current position and accordingly make a $\gamma$ ratio of the entire vocab into green tokens based on this value. This way, the green token list is not fixed, making it robust to attacks such as brute-force attacks.

Now let's check if a random text $y$ was generated from $\hat{M}$. We can measure the z-score of the token sequence $y$ and determine that $y$is watermarked if its value exceeds a threshold.

$z=\frac{|y{|}_G-\gamma n}{\sqrt{n\gamma (1-\gamma )}}$

 

The example below shows that the text generated by the watermarked LLM produces a significantly higher frequency of green tokens compared to the unwatermarked text.

 

Image 6. Example with green-red watermark applied^[9]

 

This year’s ACL 2024 also featured an LLM watermark paper, “Who Wrote this Code? Watermarking for Code Generation”^[10], in which researchers at LG AI Research participated. This was also mentioned in this tutorial when introducing various watermarking techniques. In the paper, we showed that existing watermarking techniques are prone to producing incorrect results, such as syntax errors, even if only one token is incorrect in a code generation task. Additionally, we demonstrated that it is difficult to statistically distinguish whether a watermark is present or not due to the low entropy of the logit. To address these issues, we proposed the SWEET methodology. SWEET maintains the quality of the generated code and improves detection performance by applying the watermark only if the entropy $H_t$ of the generated probability distribution at the current location exceeds a threshold $\tau$ before applying the green-red watermark.

 

Future Plan

The safety and security of LLMs, a topic that was highlighted in the ACL 2024 tutorial, is a very important topic for those of us who develop LLMs. In fact, the vulnerabilities and limitations of LLMs that were raised in the tutorial are ones that we continue to encounter as we develop LLMs. Addressing these issues well is the way to create a desirable LLM.

Recently, LG AI Research unveiled EXAONE 3.0 with superior performance and open-sourced 7.8B instruction tuned model. As disclosed in Technical Report, LG AI Research is taking data copyright and licensing issues seriously and is continuously improving the LLM by identifying vulnerabilities through various evaluation methods, such as red teaming. We will continue to pursue research and development to secure top-level global technological competitiveness and to implement ethical AI.

ACL Trends Review Series

#2. [ACL 2024] A New Approach to Chart Understanding and Reasoning
#3. [ACL 2024] Towards efficient Large Language Models
#4. [ACL 2024] AI-generated Text Detection Research Trends
#5. [ACL 2024] Research Trends in LLM Evaluation Methods for Faithfulness and LLM Efficiency
#6. [ACL 2024] Developing a Large Language Model with Cross-Cultural Considerations