[ACL 2024] Towards Efficient Large Language Models

Recent developments in large language models (LLMs) have shown remarkable performance across various domains^[3,4,5].Notable advancements include improvements in natural language understanding, generation, translation across multiple languages, and specialized applications like summarization, question answering, and code generation. Models such as OpenAI's GPT-4, Google's PaLM 2, Meta's LlaMA^[4], and LG AI Research's Exaone^[3] have demonstrated significant progress in handling complex reasoning tasks, maintaining context over long conversations, and generating coherent and contextually relevant outputs.

However, as these models have grown larger and more complex, sometimes reaching hundreds of billions of parameters, they face significant challenges related to training and inference^{[1 ,2, 6]}. The immense computational power required to train these models can be prohibitively expensive and time-consuming, often demanding specialized hardware like GPUs or TPUs running for weeks or even months. Additionally, deploying these large models in real-world applications poses its own challenges: inference can be slow and resource-intensive, making it difficult to achieve real-time performance or deploy the models on devices with limited computing power. For example, the LLaMA 405B model requires multi-node inference, necessitating a coordinated setup of multiple machines to handle the model's size and complexity. This makes inference both costly and technically challenging, further limiting the scalability and accessibility of such models. .

Consequently, ongoing research focuses on developing more efficient algorithms and architectures to make LLMs more practical and sustainable for widespread use.

In this blog, we will explore various approaches to enhancing the efficiency of large language models (LLMs), with a focus on research presented at ACL 2024. We'll highlight key papers and innovations that address how to optimize the performance and resource usage of these models, offering insights into the latest advancements in making LLMs more efficient.

1. LayerSkip: Enabling Early Exit Inference & Self-Speculative Decoding (Meta)^[1] Summary

LayerSkip is a method designed to speed up the inference of large language models (LLMs). It involves training the model with layer dropout and an early exit loss function to improve early exits. During inference, it uses a self-speculative decoding technique that exits early and then verifies results with remaining layers, reducing memory usage and enhancing efficiency.

Motivation

The motivation for this work arises from the observation that early layers in large language models (LLMs) can often produce accurate predictions. For instance, Image 1 illustrates that the tokens predicted by the lower layers of the Llama1 7B model are often correct. This indicates that the model's computational resources might be used inefficiently, as accurate predictions can sometimes be made with fewer layers than the entire model. Similar findings have been observed with GPT-2 models.

Image 1. tokens predicted at each layer of Llama1 7B model.
This shows that early layers still produce correct predictions with relatively high accuracy^[1].

Approach

The proposed approach consists of three main stages designed to improve inference efficiency for large language models (LLMs).

1. Training with Layer Dropout and Early Exit Loss: Training with Layer Dropout and Early Exit Loss involves adapting traditional dropout methods to enhance training efficiency in transformer models. Rather than applying dropout uniformly, this approach tailors dropout probabilities on a per-layer basis, varying by sample and timestep.

   In addition to layer dropout, early exit loss is introduced to improve the prediction accuracy of lower layers. During training, the model's language modeling head (LM head) is designed to process outputs from various layers. Early exit loss is applied at each layer to directly supervise the lower layers' contributions to the language modeling task. The total loss function uses a normalized per-layer loss scale, adjusted by a binary curriculum function that decides whether early exit is enabled for each layer. This scaling increases across layers, imposing greater penalties on later layers where predictions are generally easier. Notably, this approach does not add extra LM heads but utilizes a single LM head across all layers.

2. Inference with Early Exit: At inference time, the model exits early after a predetermined number of layers (E), skipping the remaining layers and directly mapping to the final output. This reduces computational costs compared to using the full model.

3. Self-Speculative Decoding: To address accuracy loss from early exits, a novel self-speculative decoding method is used. This involves generating initial predictions (drafting) with early exits and then verifying and correcting these predictions using the remaining layers. This technique reuses computations from the drafting stage to enhance efficiency and reduce memory usage.

To be specific, the draft model is the early layers of itself, and hence the hidden states mapped by the draft model could be reused when running inference with the big model (self). The process is illustrated with Image 2.

Image 2. Illustration of how self-speculation decoding proposed by the approach enables fast and efficient inference^[1]

Overall, the approach combines innovative training techniques with efficient inference strategies to speed up LLMs while maintaining accuracy.

Result

Image 3. Evaluation of Llama2 7B and Llama2 13B trained with the proposed approach^[1]

The results indicate that the proposed approach is competitive with, and in some cases surpasses, the performance of the baseline model (the original Llama model). For instance, in Code Generation tasks, the proposed method significantly outperforms the Llama2 7B baseline model.

Image 4. Speed analysis^[1]

Additionally, Image 4 illustrates the speed improvements achieved with the proposed approach. By enabling self-speculative decoding, the proposed method achieves up to a 1.83-fold increase in speed compared to the baseline, both in the CNN-DM task and in code generation.

2. MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources (Amazon)^[2] Summary

This work presents MATTER, an efficient memory-augmented transformer designed to address the limitations of traditional retrieve-and-read approaches in knowledge-intensive tasks such as question answering. MATTER enhances performance by retrieving relevant knowledge from multiple heterogeneous sources, including both unstructured (e.g., paragraphs) and semi-structured (e.g., QA pairs) sources, represented as fixed-length neural memories. This model significantly reduces computational cost and latency compared to conventional methods by handling diverse knowledge sources more effectively.

Motivation

The motivation behind this work centers on overcoming critical limitations faced by current retrieval-augmented models in natural language processing tasks, particularly in open-domain question answering (QA). These models typically follow a two-step process: first, they retrieve relevant information from a knowledge source, and second, they read and comprehend this information to generate responses. Despite their effectiveness, these models face two major challenges:

1. High Inference Latency: The performance of retrieval-augmented QA models is often hindered by significant inference latency. This problem arises because the retrieval phase can result in very large context lengths for the reading phase. For instance, models like Fusion-in-Decoder retrieve numerous documents, each with a substantial token count, leading to the reader model attending to thousands of tokens during response generation. This large context length creates a bottleneck, slowing down the processing speed. Even with smaller reader models, such as T5-base, these models struggle to handle more than 10 questions per second. This latency issue is attributed to the increased context length that the reader must process, which impacts the efficiency of the model.

2. Narrow Focus on Knowledge Sources: Existing retrieval-augmented models typically rely on a single type of knowledge source, such as unstructured data (e.g., Wikipedia paragraphs) or semi-structured data (e.g., QA pairs). Each knowledge source type has its strengths and weaknesses. Unstructured data is broad and accessible but may contain noise and lack precision, while semi-structured data is concise and clear but harder to gather. This narrow focus limits the model's ability to cover diverse knowledge effectively. Current approaches that attempt to combine knowledge sources often involve costly transformations, such as converting unstructured text into QA pairs, which can introduce noise or corrupt the information.

Approach

The MATTER approach tackles open-domain question answering by focusing on reducing inference latency and expanding the types of knowledge sources used. In traditional retrieval-augmented QA systems, the model first retrieves relevant information from a knowledge source and then generates an answer based on this information. MATTER improves on this by using neural memory to represent the retrieved knowledge instead of processing raw text, which speeds up inference significantly. Additionally, it supports multiple types of knowledge sources, such as QA pairs and paragraphs, broadening the range of information the model can utilize.

The MATTER model uses the T5-base encoder-decoder as its core reader. It encodes questions and memories separately before combining them in a final encoding step (cross-encoding in the last n layers of encoder). The neural memory representation is fixed-length and agnostic to the structure of the knowledge, making it adaptable to various knowledge types. The overall framework is depcited in Image 5.

Image 5. Overall framework of MATTER.
. As you can see from Image 5, the question is fed to both encoder and a retriever.
Once memories are retreived and question is mapped, they are cross-encoded in the remaining encoder layer.
In this image, memory size is 2 and the number of retrieval is 4^[2].

Training MATTER involves two main components: an auto-encoding loss, which ensures the neural memory accurately reflects the original knowledge, and a memory-augmented generation loss, which ensures the model generates correct answers based on the retrieved memories. For models using just QA pairs (MATTER-QA), the loss function combines these components. For models incorporating both QA pairs and paragraphs (MATTER-QA/PRG), an additional loss term is included for paragraphs. This approach not only speeds up the model but also increases its ability to handle various types of knowledge, improving both performance and versatility.

Result

Image 6. experiment results on popular open-domain QA tasks^[2]

The evaluation of the MATTER approach shows notable performance improvements and speed advantages compared to existing models. In the zero-shot setting, both MATTER-QA and MATTER-QA/PRG significantly outperform QAMAT^[7] and EMAT^[8], memory-based efficient baselines, across multiple datasets, with MATTER-QA achieving an EM score of 47.5 on the TQA dataset, compared to EMAT's 32.4 and QAMAT's 34.1. The performance gap widens further when MATTER-QA/PRG uses both QA pairs and Wikipedia paragraphs, reaching an EM score of 51.6. In the fine-tuned setting, MATTER models remain competitive, balancing high QA performance with faster inference speeds compared to models like FiD and RAG. MATTER-QA and MATTER-QA/PRG achieve EM scores of 51.2 and 56.0, respectively, while being at least 10 times faster. The MATTER-QA (Fast) variant, utilizing a smaller retriever, processes 463 questions per second with slightly reduced performance, demonstrating the model's flexibility and efficiency. Overall, MATTER models offer strong accuracy and impressive speed, outpacing existing methods in both metrics.

Conclusion

At ACL 2024, numerous influential papers have focused on improving the efficiency of LLMs. In this post, we explored two notable approaches: layer-skip and MATTER. Both approaches represent significant advancements in LLM efficiency, contributing to the ongoing evolution of language model technology.

LG AI Research also recognizes the importance of improving the efficiency of LLMs. At LG AI Research, we are dedicated to continually improving the efficiency of LLMs, including our newly released model, EXAONE 3.0. Our ongoing efforts focus on optimizing performance to better meet user needs and enhance overall model efficiency.

ACL Trends Review Series

#1. [ACL 2024] Emerging Trends and Key Insights in LLM Research
#2. [ACL 2024] A New Approach to Chart Understanding and Reasoning
#4. [ACL 2024] AI-generated Text Detection Research Trends
#5. [ACL 2024] Research Trends in LLM Evaluation Methods for Faithfulness and LLM Efficiency
#6. [ACL 2024] Developing a Large Language Model with Cross-Cultural Considerations