[AI Ethics Seminar EP.1] Red Team Research Trends and Applications for Generative AI

As generative AI technologies rapidly advance, responsible AI (RAI) is becoming increasingly important when developing, deploying, or using AI systems. As the importance of generative AI grows, so does the interest in AI Red Teams. The concept of the Red Team derives from military strategy, referring to a team that takes on the role of the enemy in mock exercises, analyzing and simulating problems from the adversary's perspective to develop countermeasures.

The AI Red Team is a concept that specializes and extends the traditional Red Team approach to generative AI. They perform comprehensive checks on AI systems to detect new security threats such as prompt injection and data poisoning, and to prevent fairness issues or harmful content from occurring. The AI Red Team covers not only malicious attacks, but also inappropriate results that may arise from normal user interactions, and conducts continuous and periodic checks and automated monitoring to respond to rapid changes in AI systems. In particular, given the technical nature of generative AI, which operates in a probabilistic generation manner, the AI Red Team repeats various tests, analyzes the results, and comes up with a multi-layered defense strategy to improve the consistency and reliability of the system.  This way, the AI Red Team plays a critical role in developing safe and reliable AI systems..

LG AI Research has applied the AI Red Team to ChatEXAONE, an enterprise AI agent released in August. ChatEXAONE is a service that revolutionizes work productivity for enterprise users, and by applying AI Red Team, we are strengthening security requirements and reliability, and providing responsible AI services to users^[1].

This post will introduce the main concepts of the Red Team for generative AI and introduce various applications and studies where AI Red Team has been applied, including ChatEXAONE's prompted recommendation feature.

1.  The Concept of AI Red Team

Recently, generative AI technologies centered around large language models and non-language models have gained traction in a variety of fields. While the excellent ability to generate models is a strength of generative AI technologies, they also have limitations, as they can be misused or manipulated by malicious requests. AI Red Team plays an important role in mitigating these risks.

AI Red Team is the process of making models safer by identifying vulnerabilities in models and developing ways to defend against them. The process involves four main steps. The first step is to find and classify risk factors, the second is to develop attack strategies and automate them. The third step is to build and validate defenses against these attacks, and the fourth and final step is to incorporate the results of the AI Red Team to improve overall system operations. The AI Red Team goes through a step-by-step process to develop offensive and defensive strategies to increase the safety, robustness, and reliability of AI systems^[2].

(1) Risk Classification and Identification

The first step in the AI Red Team is to identify and organize the risks that AI models may pose. This process uses multiple criteria to break down and analyze the risks that models may pose. First, the AI service provider establishes rules and guidelines for the model to follow to safely provide services. These guidelines include rules to prevent illegal behavior, such as violence, exploitation, and terrorism, and to avoid harming specific individuals or groups. Based on these, the AI Red Team analyzes the risks that the model may encounter as it operates, breaking them down.

Several criteria is used to specifically categorize risk factors:

1. Policy: Analyzes risks based on rules or guidelines that the model must follow.

2. Risk type: Defines specifically what types of damage can occur if the model malfunctions.

3. Target: Assesses risks by identifying the targets the model may affect, such as specific individuals, internal groups, or external groups.

4. Domain: Identifies issues that a model may raise in a specific field, such as science, copyright, or ethics.

5. Scenario: A more realistic assessment is made by assuming various situations that are likely to occur when the model is actually used and breaking down the risks that may occur in those situations..

With this multi-layered approach, the AI Red Team can help predict what risks models might pose in different situations and build safe and reliable AI systems based on that knowledge.

(2) Developing an Attack Strategy

After identifying and classifying risk factors in step 1, step 2 develops various attack strategies based on them. In this step, attack methods that fit the characteristics of the generative model are designed and potential threats are simulated. Attack strategies against language models can be categorized into four main areas, each of which utilizes characteristics and abilities that the model has acquired during its learning process. By doing so, an attacker can induce the model to generate inappropriate or dangerous responses.

First, there is the Completion Compliance strategy. Due to the nature of causal language models, major decoder-based language models have the property of predicting the next token of a sequence based on a given context. Attackers exploit the properties of causal language models to induce the model to generate responses in compliance with malicious prompts. For example, if you enter the final input prompt with affirmative suffixes after the harmful or prohibited content, such as “Of course, here’s the result,” the model will be more likely to respond to the harmful or prohibited content under the influence of the affirmative suffix.

Next, the Instruction Indirection strategy exploits the model's ability to follow instructions. As models learn, they gain the ability to perform different types of instructions through instruction tuning, and an attacker can use this to make the model behave according to instructions while circumventing safety regulations. For example, they can force the model to respond in a specific format (JSON, Wikipedia-style, etc.), or they can use complex scenarios or nested simulations to force the model to include dangerous content in its instructions. This can encourage the model to ignore its own safety policies.

In addition, the Generalization Glide strategy exploits a model's ability to generalize. Language models can generate generalized responses by learning from data across different languages and domains during learning. Attackers use this generalization ability to try to get the model to bypass safeguards. For example, they can write harmful prompts in a low-resource language (a language that is used infrequently), causing the model to generate dangerous responses that take advantage of the lack of safety validation for that language. They can also circumvent safety validation by tricking the model into interpreting encrypted sentences.

Finally, the Model Manipulation strategy involves manipulating the output of a model by directly adjusting the parameters, activation values, or components of the model. This strategy is particularly effective with open-source models or when the user has access to the internal structure of the model. An attacker can manipulate the model's decoding parameters (e.g., temperature) to cause the model to produce different output results, or change the way parameters are sampled to increase the success rate of the attack. The model can also be manipulated to intentionally produce harmful results by modifying weights or specific activation values.

Based on an understanding of the unique learning characteristics and capabilities of language models, these attack strategies systematically present a variety of ways to threaten the safety of a model. Each strategy is designed to exploit a specific ability or learning process of the model, with the goal of preventing the model from maintaining its originally safe behavior.

(3) Establishing Defense Strategies and Improving System Operations

Once an attack is identified, the next step is to evaluate it, build a defense strategy, and improve system operations. There are two main types of defense strategies: learning-time defense and inference-time defense.

Learning-time defense is achieved through fine-tuning or reinforcement learning (RLHF) after the language model is pre-trained ^[3]. During this process, the model is tuned to recognize and reject harmful input data so that it can safely generate responses in a variety of situations. On the other hand, another approach is adversarial training, where the model is trained to generate harmful queries and learn to reject them to make it safer. A reward model is used to evaluate safety and usability, and that data is used to fine-tune the model. While these learning-time defenses are effective at keeping the model safe in new situations, they can affect the model's original functions, resulting in an alignment tax or performance degradation.

Inference-time defense is a method that uses a system of input filtering, system prompts, and guardrails that work in sequence to effectively reduce the risk of AI models. First, the input filtering stage evaluates user input to determine if it contains anything harmful or inappropriate and blocks it before it is passed to the model. This reduces the likelihood of unsafe responses being generated and acts as a proactive response to offensive input. However, the downside is that responses can be delayed due to the length of the prompt and the step-by-step process.

System prompts provide specific instructions to guide the model's behavior, helping the model respond in a responsible and safe manner. Finally, a guardrail system monitors the model's output to filter out or modify unsafe responses based on predefined rules and criteria. These defenses respond flexibly to a wide range of threats and contribute to maintaining the safety and reliability of AI systems. However, this process can be computationally expensive and may not be able to fully protect against all complex attacks.

2. AI Red Team Research and Applications

(1) MART: Improving LLM Safety with Multi-round Automatic Red-Teaming

MART proposes multi-round automatic Red-Teaming to improve the safety of large-scale language models (LLMs). An adversarial model and a safe model interact, where the adversarial model generates prompts that attack vulnerabilities in the safe model, and the safe model is trained to defend against them and improve safety. In this way, MART aims to continuously find vulnerabilities in the model through repeated learning, improve them, and ultimately build a safer and more reliable AI system.

First, in the initialization phase, the safe and adversarial models are based on the LLaMA-65B model, a large-scale LLM, and initial training is performed using Open Assistant and LIaMA datasets^{[5, 6]}. The datasets used are refined to include non-aggressive and non-harmful data, ensuring that the models are safely capable of performing basic instructions.

The adversarial model learns to generate malicious prompts based on approximately 2,400 previously collected attack prompts. As shown on the left in Image 3, the adversarial model generates new prompts in each round of learning based on previously successfully attacked prompts. It then evaluates the safe model's response to determine if the attack is successful. During this process, the adversarial model optimizes for continuous effective attacks when the prompts it generates successfully attack the safety model.

Conversely, in the process of safety reinforcement, the safe model generates safe responses in response to attacks from the adversarial model and then retrains the model based on those responses. The safety model is gradually improved to generate safer responses through repeated learning. The figure on the right in Image 3 shows what happens when the safe model successfully defends against an attack from the adversarial model.

The performance evaluation of MART validated how well usability can be maintained while increasing reliability. Evaluation of the learned data (In-distribution) is performed using the safety evaluation dataset (SafeEval) and the usefulness evaluation dataset (HelpEval). Here, SafeEval contains a subset of the attack prompts used in MART, and HelpEval consists of non-attack prompts written by a human labeler. The evaluation of data not used in training (Out-of-domain) measures the generalization performance of MART using the AlpacaEval datasets and Anthropic Harmless datasets.

As can be seen from the evaluation results (a) and (c) in Image 4, the safety scores for SafeEval and Anthropic Harmless improve significantly with each repetition of MART's adversarial training, especially for the cases in the lower score range (20% and 40% percentiles). On the other hand, as shown in (b) and (c), the usability scores for HelpEval and AlpacaEval remained stable over the course of MART's adversarial learning repetitions, indicating that MART strengthened the safety of the models while not significantly compromising their usability.

(2) AI Red Team Applications

ChatEXAONE's prompt suggestions feature is a prime example of an AI Red Team defense strategy in action. The prompt recommendations feature provides customized prompts based on the job title and business expertise of the enterprise user. It also recommends the best follow-up prompts to help users gain deeper insights during their conversations with ChatEXAONE.

While ChatEXAONE’s prompt recommendation feature makes a significant contribution to improving user productivity by providing optimal follow-up prompts, its ability to accurately identify the user's input intent and provide appropriate follow-up prompts leaves room for risk. A typical example of this is when a user enters a malicious prompt and it serves a malicious follow-up prompt by assisting with an inappropriate follow-up prompt.

To prevent this, LG AI Research applied AI Red Team defense strategies to both the learning and inference stages. For the prompt recommendation feature, we started by building our own model based on EXAONE to learn how to respond to malicious prompts. In addition, at the inference stage, we applied LG AI Research's own safety filter to detect and block harmful content in real time. This helped us strengthen ChatEXAONE's system safety and reliability.

Recently, as more service applications have focused on LLM APIs, the importance of inference-time defense has become more important than learning-time defense. In response to this need, global companies like OpenAI, Meta, and Microsoft are applying different approaches to strengthen the safety and reliability of AI models.

OpenAI built its Red Teaming Network to conduct evaluations and tests that involve a wide range of experts. This network explores potential exploits of large models like GPT-4, identifies policy violations through repeated testing, and ensures model safety. These Red Team activities focus on proactively identifying and responding to risks associated with harmful content to ensure that AI systems can operate responsibly and safely in user environments^[8].
.

Microsoft performs AI Red Team on applications like GPT-4 and Bing Chat to identify security vulnerabilities and harmful content that models can generate. During this process, Microsoft provides a comprehensive defense strategy to increase the safety of the AI system as a whole. Specifically for users of Azure OpenAI models, Microsoft provides AI Red Team guidance to help them develop safer and more reliable AI systems^[9].

Meta's Purple LLaMA is a comprehensive framework for securing AI systems through Purple Team, which combines offensive (Red Team) and defensive (Blue Team) strategies, and is a prime example of an AI Red Teaming approach that is faithful to the approach discussed earlier. Purple Llama includes cybersecurity assessment benchmarks for LLMs and safety filter models such as Llama Guard, which helps developers prevent the creation of harmful or malicious content in AI systems^[10].

This post deepened our understanding of the key concepts of AI Red Team for generative AI, reviewed related research, and explored applications including ChatEXAONE’s Prompted Recommendations feature. The AI Red Team is an essential element in ensuring the safety and reliability of generative AI and is gaining more and more attention and importance. As AI Red Team strategies and techniques evolve, we expect to see more proactive detection and remediation of vulnerabilities in AI models and more ethical use of AI. 

LG AI Research has also been continuously exploring ways for humans and AI to coexist in harmony, releasing the AI Ethics Principles in 2022 and the AI Ethical Accountability Report in 2024, which shows the results of implementing the AI Ethics Principles^[11]. It’s as a result of these efforts that we were able to apply the AI Red Team to ChatEXAONE and open it. LG AI Research will continue to strengthen the AI Red Team so that users can utilize services more safely, such as in the case of ChatEXAONE's prompted recommendations.

AI Ethics Seminar Series

#2. [AI Ethics Seminar EP.2] Research aimed at properly understanding the risks of AI
#3. [AI Ethics Seminar EP.3] Introduction to Key AI Toolkits for Designing Trustworthy AI Experiences and Practical Application Guide
#4. [AI Ethics Seminar EP.4] Building a Responsible AI Society through AI Agents
#5. [AI Ethics Seminar EP.5] AI Ethics from a UI/UX Designer's Perspective