[ICLR 2024] Compression of Neural Networks Research Trends

Introduction

Deep neural networks (DNNs) models, which learn complex non-linear relationships in data through deep structure, exhibit excellent performance but also require a lot of computational resources and memory. To address these drawbacks, research on model compression techniques has long been an important focus. Model compression is essential for the utilization of DNNs, especially in resource-constrained environments like mobile devices.

Recently, the emergence of Large Language Models (LLMs) has further emphasized the importance of model compression techniques, as LLMs are known for their vast size and substantial computational requirements. Model compression research aims to reduce the size and computational complexity of models while maintaining or improving performance. This is expected to make DNNs more effective in a wider range of applications. This blog post will introduce the papers presented by LG AI Research at ICLR 2024, and take a look at two studies on compression: one for image tasks and one LLM tasks.

LG AI Research’s Data Intelligence (DI) Lab presented “LESS: LEARNING TO SELECT A STRUCTURED ARCHITECTURE OVER FILTER PRUNING AND LOW-RANK DECOMPOSITION^[1]” at “ICLR 2024 The 5th Workshop on Practical ML for Low Resource Settings (PML4LRS).” In addition, LG’s researchers discussed the current state of research on Lightweight Neural Networks and the way forward with researchers from Qualcomm and other companies. This post will introduce the papers presented by LG AI Research and the important insights gained from the conference, along with a review of two additional papers.

1. LESS: LEARNING TO SELECT A STRUCTURED ARCHITECTURE OVER FILTER PRUNING AND LOW-RANK DECOMPOSITION

Image 1. Comparison of the highest performance of low-rank compression,
pruning, and hybrid methods according to compression ratio.
The X-axis represents FLOPs, and the blue represents performance when applying our method^[1]

Filter pruning and low-rank decomposition are both techniques for compressing neural network models, but they are based on different structural assumptions. Filter pruning assumes that some filters are redundant and removes them on a filter-by-filter basis, while low-rank decomposition assumes that certain dimensions of the weight matrix are less important and reduces the rank of the tensor to compress the model. However, existing research has not effectively combined these two techniques, as shown in Image 1, resulting in only marginal performance improvements compared to using each method alone.

Accordingly, this paper studied how to efficiently integrate filter pruning and low-rank decomposition. In particular, it was deemed reasonable decided that a learning-based method would be neccessary to select appropriate ranks without heavily relying on heuristics. To this end, and threshold learning, a new SGD-based learning technique for rank selection, was proposed. In this paper, the existing mask learning-based filter pruning method was improved and used. This allowed for the creating differentiable modules that can be trained end-to-end.

Image 2. Overview of the forward process of the LeSS algorithm^[1]

The proposed method (Image 2) first rearranges the weight tensor, then performs SVD, and learns a scalar variable whose outliers can be controlled by a threshold. This scalar variable is optimized learns to minimize the cross-entropy loss while still satisfying the target constraints. For filter pruning, a method of learning a mask by placing a gate on each channel was used, where the mask variable also optimized to satisfy the target constraints while minimizing the cross-entropy loss. In particular, a scheduled sigmoid function was used in this paper to gradually drive the mask values to have extremes values close to 0 or 1, and the sum of all gates was used to adjust the sparsity to be close to the desired budget.

The proposed LeSS method outperformed existing pruning, low-rank decomposition, and hybrid methods by a large margin. LeSS outperformed the baseline model while reducing the model’s FLOPs by 80% in the result of ResNet56 on CIFAR10. On the ImageNet dataset, it maintained similar performance to the baseline while reducing FLOPs by 65%. In additional experiments on various network structures, LeSS consistently demonstrated superior compression performance. Notably, when applied to MobileNetV2, a well-known lightweight model, it achieved higher performance than the baseline while reducing the number of parameters by 36% and FLOPs by 35%.

LeSS is designed to avoid heuristics and use a differentiable approach to efficiently solve combinatorial optimization problems (COPs). Without the need for a special regularizer that takes compression into account, efficient differential learning to select filters and ranks can be used to create a network that satisfies the desired target resource without iteration. This paper experimentally demonstrated the effectiveness of this method specifically forlimited to vision tasks, but it is believed that it can be effectively applied to other tasks as well.

2. SHEARED LLAMA: ACCELERATING LANGUAGE MODEL PRE-TRAINING VIA STRUCTURED PRUNING^[2]

Structured pruning is a methodology that removes groups of parameters in a model to compress it and speed up inference. However, existing structured pruning methods often produce unusual model configurations that deviate from widely used architectures. For example, they may create models with a different number of heads per layer, which causes overhead in inference compared to a typical uniformly sized layer configuration^[3]. This study aims to prune the source model to satisfy all the target structures specified earlier.

The contributions of this paper can be summarized as follows. (1) For the first time in the LLM field, a pre-defined architecture was selected as the target model and the current architecture was pruned to have the same configuration. (2) During pruning, constraints on the shape of the pruned model were directly imposed using Lagrange multipliers to ensure the shape of the target model was maintained. (3) Finally, dynamic batch loading was used to address the performance degradation that may occur when using the same learning data distribution for the pruned model as was used for the pre-trained model.

Image 3. Schematic diagram of the operation of the targeted structure pruning method.
Light colors indicate pruned substructure^[2]

More specifically, compressing LLM models which are based on transformer architectures is challenging, as they need to reduce all dimensions and structures of varying granularity. To address this, this paper uses the configuration of existing pre-trained models as the target architecture. This is based on the intuition that these pre-defined configurations already well optimize the trade-off between expressiveness and inference efficiency of the model. For example, to create a 2.7B model, the INCITE-Base-3B^[4] architecture is used as the target structure.

In this paper, pruning masks are trained for model parameters at different granularities, ranging from global, such as layers that persist across all layers and hidden dimensions, to local, such as attention heads and intermediate dimensions. Each mask variable controls whether to prune or retain the associated substructure. For example, if z_layer is 0 for a given layer, that layer is removed. Image 3 shows an example of how the pruning mask controls the pruning structure.

 

${\tilde{\mathcal{L}}}^{\text{head}}(\lambda ,\varphi ,z)={\lambda }^{\text{head}}\cdot (\sum z^{\text{head}}-H_{\mathcal{T}})+{\varphi }^{\text{head}}\cdot {(\sum z^{\text{head}}-H_{\mathcal{T}})}^2.$

As shown in the equation above, this paper configures the structural information of the target model directly into the loss function to explicitly learn to fit the model structure. 

The overall loss used when learning is as follows. It can be confirmed that it is configured by including losses for all granularities (head, int, layer, hidden).

 

${\mathcal{L}}_{\text{prune}}(\theta ,z,\lambda ,\varphi )=\mathcal{L}(\theta ,z)+\sum _{j=1}^{L_{\mathcal{S}}}{\tilde{\mathcal{L}}}_j^{\text{head}}+\sum _{j=1}^{L_{\mathcal{S}}}{\tilde{\mathcal{L}}}_j^{\text{int}}+{\tilde{\mathcal{L}}}^{\text{layer}}+{\tilde{\mathcal{L}}}^{\text{hidden}}$

 

Finally, performance was improved by dynamically changing the proportion of data based on the difference in loss between pre-trained and pruned models through dynamic batch loading. In particular, this method has the advantage of being an on-the-fly method that can be utilized even for open-source datasets without data constraints. The method presented in this paper was evaluated on various datasets, and its superiority was confirmed (Image 4).

Image 4. Performance result table of Sheared-LLaMA^[2]

This study utilized structured pruning techniques to compress language models and improve inference speed. Existing structured pruning methods often have the limitation of creating abnormal model configurations, so to overcome this, an approach that prunes the source model to fit a predefined target structure was proposed. The approach in this study is significant in that it overcomes the limitations of existing structured pruning techniques and provides a new direction to effectively compress language models and increase inference speed. The proposed methodology is expected to enable the efficient deployment and utilization of LLMs in real-world applications. Future research will aim to further develop the ideas presented in this study, applying them to various architectures and datasets to verify and generalize their effectiveness.

3. SPARSE MODEL SOUPS: A RECIPE FOR IMPROVED PRUNING VIA MODEL AVERAGING^[5]

This paper proposed a new compression paradigm by combining the pruning method with the ensemble method. As previously mentioned, pruning has long been a popular technique for removing unnecessary parameters. The main advantage of this method is that it reduces both parameter storage space and computation while maintaining similar to that of the pre-compression model. In addition, methods to improve prediction performance through ensembles of multiple models have been proposed^[6]. In particular, recent research has focused on utilizing ensembles of pruned models to reduce the inference cost of ensemble models. For instance, the Model Soups method^[7], which averages all parameters to create a single model, has been proposed. However, this method suffers from the problem that averaging all parameters may reduce the overall sparsity. To address this, the study proposes a method to retrain the pruned model to generate a model suitable for averaging. This is accomplished by exploring different hyperparameter configurations, similar to the finetuning step of transfer learning. The contributions of this paper are as follows.

1. It was confirmed that pruned models can be retrained with different hyperparameters and averaged to create a model with better generalization and OOD performance than individual models.

2. The proposed Sparse Model Soups (SMS) method proposes the idea of starting each pruning-retraining step from the average model.

3. The SMS method was integrated with various state-of-the-art techniques to improve pruning performance during training.

Image 5. (Left) Schematic diagram of the SMS algorithm. (Right) SMS numeric code^[5]

This study proposes a very simple method: repeating prune-retrain-merge, as shown in Image 5 (right), to improve performance while reducing the number of parameters. The peculiarity here is that it starts from the same model, differentiates the seed, weight decay, retraining epochs, learning rate, etc. to obtain m models with the same structure but learned to have different characteristics. These models are then merged into a single model by averaging their parameters. This method has the advantage of maintaining the same pruning ratio after merging into a single model as retraining occurs without structural changes in the models, which that are all pruned with the same structure. In particular, it has a great advantage in terms of learning speed as m models can be trained in parallel. Another great advantage is that this method can be applied to pre-trained models, without having to train a model from scratch. When averaging the parameters, each model was either given an equal weight of 1/m and used as the average value (Uniform Soup), or the models were listed according to their verification accuracy, and only those with higher verification accuracy than the previous (Greedy Soup) were averaged.

Image 6. CIFAR-100 performance of WideResNet-20 model
(a) Performance comparison according to various sparsities of Uniform Soup and Greedy Soup
(b) Performance comparison according to various retraining epochs^[5]

This study applied the methodology to a vision task to demonstrate its effectiveness. First of all, when comparing the method of averaging m models uniformly and the method of averaging only models that showed performance improvement, there appears to be no significant difference. Second, the performance saturates when the number of retraining epochs is 40 or more.

Image 7. Performance results using SMS to the ResNet-50 model for the ImageNet dataset^[5]

The results of applying it to ImageNet show that the performance improvement over previous studies is consistent, especially when combined orthogonally with other existing methods^[5].

The significance of the proposed method in this study is that it combines the advantages of existing pruning and ensemble techniques, presenting a new approach that can simultaneously achieve model compression and performance improvement. In particular, this paper is significant in that it efficiently utilizes model soups, which have been actively studied in recent research to effectively compress LoRA modules in LLM. This is expected to enable efficient distribution and use of large language models when applied to other domains such as LLM. In future studies, the proposed method will be applied to various model architectures and datasets to verify its effectiveness and develop it into a more advanced model compression technique.

Additionally, through in-depth discussions with corporate researchers at ICLR 2024, it was agreed that research on language model (LLM) compressionresearch has been very active in recent years. Most notably, insights from methodologies previously studied in vision and audio are being used to adapted them to the unique characteristics of LLM models. For example, traditional model compression techniques such as knowledge distillation are still in use, and new techniques are being explored to optimize the interaction between teacher and student models. These methods have gained a lot of attention as they can significantly reduce the number of parameters while maintaining the performance of the model.

It was also confirmed that quantization and low-precision computation techniques are playing an important role in language model compression. In particular, recent research has shown that low-precision quantization of 8 bits or less can be performed effectively without degrading the performance of the model. These techniques are expected to make a significant contribution to the distribution and real-time responsiveness of large language models.

Furthermore, there has been active discussion on how distributed training and model parallelism techniques can be leveraged to train and operate large language models more efficiently. Such techniques could provide very useful solutions for companies operating large language models, especially in the cloud environment.

Conclusion

Compression technology for deep neural networks (DNNs) is an essential area of research for the practical application and commercialization of these innovative language models. This is because model compression can provide important benefits such as reduced memory usage, faster inference, and lower deployment costs - all of which are critical to operating large-scale models.

Lightweight models also enable excellent performance, especially in resource-constrained environments such as mobile devices and embedded systems. This can greatly expand the potential for applications in diverse areas as real-time voice recognition, augmented reality, and autonomous driving. These applications require fast and efficient inference, and lightweight models can provide performance similar to high-performance server-based models while minimizing resource consumption to maximize efficiency. In addition, model compression offers has the economic benefit of reducing deployment costs. Storage space and transmission bandwidth are major cost factors when running large-scale models in a cloud environment, and compression technology can effectively reduce these costs. This can directly contribute to lower operating costs for companies. Furthermore, lightweight models can operate well in unstable or limited network environments, making services more accessible to a wider range of users.

Taken together, model compression research for deep neural networks, including LLMs, is a very important research area that goes beyond simple technological advancements and enables innovations in real-world applications. If various studies related to language model compression go beyond simple theoretical approaches and focus on experimental verification and application cases to increase the applicability in real-world industries, this will make language model compression technology more practical and effective in the future. LG AI Research’s Data Intelligence Lab will continue to conduct research and development in this area and create more diverse and innovative use cases.

 

▶[ICLR 2024]The Latest Research Trends of Foundation Model