[ACL 2024] AI-generated Text Detection Research Trends

This post was written by Hyeongu Yun from EXAONE Lab, using EXAONE 3.0 7.8B Instruct-tuned Model. EXAONE 3.0 was utilized to simplify complex technical content and enhance sentence clarity. The model-generated suggestions were reviewed by a human to ensure accuracy and coherence.

Large Language Models (LLMs), which can generate human-like text by learning vast amounts of natural language data, have made rapid progress since 2020, opening up new possibilities for the use of AI and generative AI. LG AI Research recently released EXAONE 3.0^[1], which was developed with our own technology from the pre-learning stage of the LLM to system deployment, demonstrating our global competitiveness.

Combined with multimodal modules, LLMs are demonstrating their ability to solve previously challenging real-world problems, such as those involving Large Vision Models and web search engines, thereby enabling Retrieval Augmented Generation (RAG) systems. These models are capable of generating human-like text fluency and are leading to innovative advances in fields such as information provision, creative fields, customer service, and others.

However, with great power comes great responsibility. As LLMs become increasingly utilized and offer more possibilities, they also bring greater concerns. While they can provide fluent and compelling answers to almost any question that their computing power allows, they can also misrepresent facts or mishandle ethically sensitive topics. This can lead to the spread of misinformation or the creation of inappropriate content, and there is also the risk that it can be used maliciously.

To reduce the potential risks of LLM, technology that effectively detects machine-generated text (MGT) is becoming increasingly important. AI-generated text detection technologies are expected to play an important role in setting ethical standards, managing data, protecting users, and more. At ACL 2024, the top-tier international conference on natural language processing, held in Bangkok, Thailand in August 2024, many researchers presented techniques for detecting AI-generated text.

This article provides a general summary of the AI-generated text detection techniques presented at ACL 2024. The papers range from studies that address limitations of existing detection techniques and achieve higher performance, to studies that take insights from classical natural language processing and present a completely new approach, to studies that present new benchmarks to quantitatively measure the performance of AI-generated text detection models.

 

1. Existing AI-generated text detection techniques and studies that complement them

One of the most successful existing detection techniques is DetectGPT^[2]. DetectGPT is based on the observation that when AI-generated text is perturbed in small amounts, the probability values over the language model for the perturbed text have a negative curvature (Image 1). In contrast, human-written text does not exhibit this tendency when perturbed in small amounts.

Image 1. AI-generated text has a negative curvature in the probability values above the language model as it is slightly perturbed,
but human-written text has a relatively weak tendency to do so^[2].

DetectGPT uses this observed trend. It computes the log-likelihoods of the language model for the suspect text and slightly perturbed versions of the text and analyzes the log-likelihood difference (Perturbation Discrepancy Gap) between the original text and the perturbed text. The larger the log-likelihood difference, the more likely it is that the text was generated by an AI such as a large-scale language model. DetectGPT has the advantage of being able to detect whether a text was generated by a given model using only the log-likelihood information of a given model without additional training on a specific model. In addition, the effectiveness of the method using the log-likelihood values of slightly modified target texts on a language model through high detection performance has been experimentally demonstrated.

However, other recent studies have reported that methodologies for detecting AI-generated text with slight perturbations have difficulty detecting adversarial perturbations ^[3,4]. For example, it has been shown that AI-generated sentences such as “The capital of South Korea is Seoul” can be detected well, but if a very slight “adversarial” perturbation is made to this sentence, such as “The capital of South Korea is Hanseong,” existing detection models will recognize it as human-written.

A paper presented at ACL 2024, “Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better”^[5] also points out the limitations of DetectGPT. The authors argue that DetectGPT’s random masking technique does not protect important tokens (words or phrases), which inevitably leads to corruption or removal of semantically significant parts, which can lead to missing important information during the detection process and thus degrade the detection performance. Furthermore, they argue that DetectGPT relies solely on the log-likelihood difference between the post-perturbation text and the original text, which is an oversimplified approach that fails to sufficiently reflect the different situations in which perturbed text may be treated differently by the model.

To compensate for this, the paper proposes “selective perturbation.” The basic idea is to strategically place masks by adjusting the probability of selecting a mask according to the importance of a text token so that nothing of importance is missed. Therefore, the task of evaluating the importance of token units must be preceded, and to do this, the authors extend the YAKE (Yet Another Keyword Extractor) algorithm^[6], which determines word importance by using the fact that the importance of a particular word is inversely proportional to the lexical richness of the surrounding words, to the token level. The authors argue that through “selective perturbation,” only tokens with token importance below a certain threshold are masked, so contextually important information is not missed (Imgae 2).

Image 2. In the Pecola technique using selective perturbation, words that are important (green background) are not perturbed.
The words in orange are the words that are perturbed^[5].

The authors go one step further than the “selective perturbation technique” and propose a detection model that distinguishes AI-generated text from human-written text using “a model that utilizes contrastive learning techniques,” which they collectively call the Perturbation based Contrastive Learning model (PeCoLa).

The detection model is trained in the following two steps:

1. Reusing importance using YAKE to reconstruct features so that important tokens carry more weight in model training

2. Applying multi-pairwise contrastive learning to analyze correlations between different pairs of inputs so that the model can better understand the nuances between texts

The authors’ methodology appears to have attempted to optimize the contrastive loss objective to improve text classification, while also including token importance directly as a feature to capture important information during training. Finally, the objective of combining cross entropy loss and contrastive loss was used to achieve stable training, allowing the PECOLA model to accurately distinguish between machine-generated and human-written text with a small amount of training data. This technique is expected to be useful for efficiently performing text generation and classification tasks, especially in limited data environments.

The experimental results described by the authors are interpreted as reflecting the authors’ intentions well. In test sets such as the Human ChatGPT Comparison Corpus (HC3)^[7], it shows an accuracy of 99.15% (few-shot setting; 512-shots), and in other test sets, it shows superior detection accuracy to DetectGPT. In particular, the detection accuracy increases linearly as the number of shots increases in a few-shot test environment, which is likely due to the influence of contrastive learning and model-based detection. The PeCoLa methodology presented in this paper is thought to have great advantages in that it shows high detection accuracy in various domains and limited data situations, and leads to more reliable prediction results by reducing unnecessary noise and increasing the robustness of the model.

A study was also presented at ACL 2024 that attempts to address the problem of “adversarial perturbations” that existing AI-generated text detection techniques are vulnerable to from a model architecture perspective. The paper “Are AI-Generated Text Detectors Robust to Adversarial Perturbations?^[4]” presents a detection model that is robust to perturbations by introducing a Siamese auto-encoder structure.

The main advantage of the Siamese Denoising auto-encoder structure is its ability to focus on essential information while remaining robust to noise. Typically trained using a combination of reconstruction loss and symmetric KL divergence loss, the Siamese auto-encoder has the advantage of being able to reconstruct consistent features from different noises because it minimizes the difference between the outputs obtained independently from the twin structure. The authors of this paper propose a detection model robust to adversarial perturbations using this advantage of the Siamese auto-encoder, which they call the Siamese Calibrated Reconstruction Network (SCRN) (Image 3).

Image 3. Schematic of the SCRN model, which is robust to adversarial perturbation,
by borrowing the Siamese Auto-Encoder structure with a twin structure. Excerpt from the SCRN paper^[4]

SCRN learns a representation that is robust to perturbations while preserving the semantic representation through a reconstruction process that adds Gaussian noise to the text and removes it. The authors claim that this allows them to reliably detect AI-generated text even in the face of perturbations such as simple token transformations or misspellings. Furthermore, the proposed model is trained to make consistent predictions under different noise conditions by utilizing the Siamese calibration technique, which encourages the model to focus more on high-dimensional contextual features. As a result, SCRN outperforms existing methods on multiple datasets and different attack scenarios, significantly increasing the reliability of AIGT detection and experimentally demonstrating its effectiveness in real-world applications.

In the experimental results described by the authors, competing models had attack success rates of about 15% to 19% (the lower the more robust the model) under various “adversarial perturbation” attacks, while SCRN had attack success rates of about 3% to 9%. SCRN also demonstrated strong generalization performance, with an attack success rate of only about 24% in experiments across different domains and genres from the data it was trained on. Even in a general environment without “adversarial perturbation,” it achieved a high detection accuracy of 99.78% on the HC3 test set, proving that the Siamese auto-encoder structure is a suitable structure for AI-generated text detection.

2. Research on AI-generated text detection inspired by classical NLP methodologies

In classical natural language processing research, many attempts have been made to structure and process human-written natural language documents, interpreting them based on their grammatical structure and the relationships between sentences. A prominent example of this is the Rhetorical Structure Theory (RST)^[8]. RST is a theory for describing the structure of texts, mainly utilized in the fields of discourse analysis and text generation, and explains how texts maintain coherence through hierarchical and connected structures. At ACL 2024, a very interesting study was presented on detecting AI-generated texts through the differences that occur when discourse structure analysis is applied to human-written and AI-generated texts, respectively.

The paper “Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs^[9]” starts from the assumption that the process humans go through when writing is fundamentally different from the process LLMs go through when generating text, starting with discourse structure. Human writers organize text through complex and diverse hierarchical structures, which are logically connected and developed on a sentence or paragraph-by-paragraph level. These structures naturally change with context and purpose, giving text deep meaning and variation, whereas LLMs are structurally simple and predictable because they create consistent patterns by building on previous tokens to predict the next one. The authors hypothesize that this difference stems from the fundamental gap between the creative and multifaceted thought process of humans and the next-token prediction algorithm of LLMs (Image 4).

Image 4. Human writers and LLMs have different ways of generating texts, even when the topic is the same^[9].

This paper attempts to analyze discourse structure to capture structural differences between human-written and AI-generated texts. First, the text is parsed hierarchically according to the Rhetorical Structure Theory (RST) to generate a tree. This tree reflects the logical structure of each text segment, and these relationships are represented in a tree structure. This hierarchical tree is then converted into a recursive hypergraph, which allows the representation of the relationships between multiple nodes simultaneously. Next, the generated recursive hypergraph is analyzed for specific discourse motifs. Human-written text usually has complex motifs, while AI-generated text is characterized by prominent repetitive patterns. We use this difference to extract features that distinguish text writers. The extracted motifs are used as input to a classification model, which effectively distinguishes between human and machine-written texts. In particular, the authors argue that this approach analyzes structural features rather than surface-level features, which enables it to perform better across various domains and paraphrased texts..

AI-generated detection techniques that utilize extracted motifs perform well on the HC3 test set, with an F1 score of 98%. In particular, in the Deepfake Text Detect test set^[10], which detects AI-generated text from 10 different domains, the detection model that adds motif analysis recorded an F1 score of 82%, compared to 74% for the model without motif analysis. The authors also ran an experiment to analyze the correlation of the formality score of the text with hyperedges (edges connecting two or more discourse nodes; expressing high-level abstract relationships) and found a correlation of 0.39 for human-written texts, but almost no correlation at 0.08 for AI-generated texts. This implies a very interesting conclusion: the texts generated by LLM are more formally organized, but have less structured discourse patterns.

3. Newly-proposed benchmarks for AI-generated text detection

The field of AI-generated text detection is receiving increasing research attention in parallel with the development of LLMs. As the performance of detection techniques gets better and better, the demand for more difficult problems seems to grow. Several new benchmark studies in this area were presented at ACL 2024, and here are three of the most prominent ones.

1) M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection^[11]

Existing research has mainly approached the problem of AI-generated text detection as a binary classification problem, and tends to focus on English. In other words, there has been little research on the problem of detecting mixed human-written and AI-generated content. To overcome these limitations, this paper proposes a multi-lingual, multi-way, multi-generator-based three-task system called M4GT-Bench, which is based on three different tasks:

1. A binary classification task that separates human-written text from AI-generated text

2. A multi-classification task that separates human-written text from AI-generated text

3. A task that identifies the boundaries when human-written and AI-generated text are mixed

In experiments with the latest detection models, it was found that they often performed poorly on new out-of-domain data. In particular, the Transformer-based models RoBERTa and XLM-R performed excellently overall, but their ability to generalize to new domains or new generator models was still limited. Furthermore, it was nearly impossible for humans to distinguish between text written by different generators.

2) RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors^[12]

Existing detection models have not been sufficiently validated on common benchmarks and have been shown to lack robustness, especially against modified sampling strategies or malicious attacks. The authors propose a large-scale dataset called Robust AI Detection (RAID) to address this issue. The RAID dataset contains more than 6 million texts, covering a wide range of variables that previous studies have not addressed: different domains, generation models, adversarial attacks, decoding strategies, etc.

The paper also presents the results of an evaluation of 12 currently existing AI-generated detection models using the RAID dataset, which showed that most detection models are vulnerable to new text generators or modified sampling strategies, and that even simple changes such as repetition penalties can significantly reduce detection accuracy.

In these experiments, when evaluated on the RAID dataset, the detectors showed a sharp drop in accuracy at a false positive rate of 5%. In particular, the accuracy of the detector dropped by up to 32 points when repetition penalties were applied, suggesting that existing detectors are only effective under very limited conditions.

3) MAGE: Machine-generated Text Detection in the Wild^[10]

This paper aimed to address the problem of AI text detection in real-world settings by comprehensively covering text generated from multiple domains and different LLMs. The authors state that the core of the paper is that they built a machine-generated text detection (MAGE) testbed to collect and analyze text from multiple domains and different LLMs. The experimental results showed that although existing detection methods degraded in performance as the diversity of domains and models increased, they still showed a certain level of detectability. In particular, they demonstrated that even 0.1% of domain data can significantly improve detector performance. The best-performing detector successfully identified 86.54% of the text generated by the new LLMs, suggesting that it is applicable to real-world application scenarios.

In conclusion, this study demonstrates that complex text generated from different domains and LLMs can be detected, and shows promise for detection in “wild” environments. It also provides important insights into future research directions by suggesting how to utilize data within some domains to improve the performance of detection models. This paper points out the current limitations of AI-generated text detection and contributes to laying out a roadmap for future research.

AI-generated text detection must go beyond solving technical problems, and focus on social responsibility. AI-generated text detection technology should be able to prevent the spread of inappropriate information and contribute to user protection and data management. We agree that it is necessary to recognize the potential risks posed by AI technologies, including AI-generated text detection technology, and to make efforts to prevent these risks in advance. We are making various efforts to comply with the “LG AI Ethics Principles” consisting of five core values: humanity, fairness, stability, accountability, and transparency^[13]. In February 2024, we also published the "LG AI Ethics Accountability Report," which highlights the achievements from one year of implementing the "LG AI Ethics Principles." We will continue to conduct research and development in a way that maximizes the benefits of AI while reducing its potential risks.

ACL Trends Review Series

#1. [ACL 2024] Emerging Trends and Key Insights in LLM Research
#2. [ACL 2024] A New Approach to Chart Understanding and Reasoning
#3. [ACL 2024] Towards efficient Large Language Models
#5. [ACL 2024] Research Trends in LLM Evaluation Methods for Faithfulness and LLM Efficiency
#6. [ACL 2024] Developing a Large Language Model with Cross-Cultural Considerations