[ICML 2022] Part 1: Long-Tail Distribution Learning

ICML 2022 (International Conference on Machine Learning 2022) is a large-scale international academic conference marking its 39th year this year. It has a citation index of 20% and an impact factor of 6.99. Participated by more than 70,000 people every year, it is deemed to be one of the most influential AI academic conferences. LG AI Research also took part in the ICML 2022 held both online and offline in Baltimore, Maryland from July 17 to 23. Three researchers from LG AI Research have selected meaningful studies that were shared at the conference. - Part 1: Long-Tail Distribution Learning – Suhee Yoon, Applied AI Research Lab - Part 2: Self-supervised learning – Hyemin Jung, Applied AI Research Lab - Part 3: Neural Combinatorial Optimization studies – Hanseul Jeong, Applied AI Research Lab

Introduction

The Applied AI Research Lab of LG AI Research aims to find meaningful information from large-scale datasets collected in real-time and provide optimized solutions to real industrial sites. We are actively conducting research by defining the significant problems in real industries and developing a data-driven model considering various real-world constraints. Long-Tail Distribution (LTD), which commonly appears in real-world, has been recognized as an important research field and various attempts have been proposed to solve this problem.

In this article, we will examine the LTD problem and solutions from traditional methods to newly proposed methods focusing on the deep learning model. After this, we will also introduce Partial and Asymmetric Supervised Contrastive Learning (PASCL) paper that was recently presented at ICML 2022.

Long-Tail Distribution

What is Long-Tail Distribution?

Long-Tail Distribution (LTD) refers to the problem in which the distribution of a dataset is skewed. The distribution of large-scale datasets collected in the real-world can be drawn to show that it has a “long-tail” as in Figure 1 below. If a database with labels for each class has LTD, it is a special case where the imbalance ratio of class imbalance is extreme. The class with dominant sample numbers is called the head class, and that with a scarce number of samples is called the long-tail class. As shown in Figure 2, it appears commonly in a wide range of domains. Typical examples include defect detection, disease detection, self-driving car accident detection, and cyber security.

Figure 1. LTD Example

Figure 2. LTD Examples in Various Domains

In order for a model to show good performance in various tasks, it is important to learn the representation of each class well. Regardless of LTD, the head class and tail class both have non-overlapped distribution. If the features of each class can be learned by differentiating from one another, it will not cause a drop in performance for the model. However, in most cases, as models learn with a bias toward the head class, there is a severe drop in performance in the tail-class.

Various methodologies were presented to make improvements to this as shown in Figure 3. Among them, we will introduce Re-Sampling, Cost-Sensitive Learning, Transfer Learning, Representation Learning, and Decoupled Training.

Figure 3. LTD Learning Categorization

Re-Sampling

1. Under-Sampling

Random Under-Sampling (RUS) is a data-level approach that randomly selects head-class samples and removes remaining datasets to match the number of tail-class samples. If each of the head-class samples contain important information, this method can result in severe loss of information and cause underfitting.

To supplement this, methods such as one-sided selection and Wilson’s editing that can select only meaningful samples representing redundant information using K-nearest neighbors (K-NN) classifier were proposed.

2. Over-Sampling

Random Over-Sampling (ROS) is also a data-level approach that randomly reproduces tail-class samples to match the number of head-class samples. However, this simple reproduction method has limitations in extracting the robust and generalizable features of tail-classes, which can result in overfitting.

To supplement this, Synthetic Minority Over-Sampling Technique (SMOTE) was proposed. SMOTE creates a new artificial sample through interpolation between each existing head-class sample. Borderline-SMOTE, Safe-Level-SMOTE, SMOTEBoost, etc., which are methods developed based on SMOTE, as well as methods developed based on Generative Adversarial Network (GAN) were proposed afterward.

Figure 4. Under-Sampling & Over-Sampling Examples

Cost-Sensitive Learning

Cost-Sensitive Learning is an algorithm-level approach that gives different loss values per class in the learning stage to re-balance. During loss design, a penalty is given to the head-class or weight is added to the tail-class to enhance the generalization performance of the model. Table 1 below is an equation that summarizes the recently proposed Cost-Sensitive Learning loss. $z$ is the predicted logit, $p$ is softmax probability, $n$ is the total number of learning samples, $n_y$ is the total number of samples of y class, $w$ is class-wise weight, $\Delta$ is class-wise margin, and $\gamma$ is loss-related parameter.

Table 1. Cost-Sensitive Learning Loss Summary

Weighted softmax loss, focal loss, class-balanced loss, balanced softmax loss, and equalization loss all correspond to the re-weighting method. LDAM loss corresponds to re-margining that adjusts the minimum margin of a model and the learned features for different classes for supplementation.

Transfer Learning

In LTD, there is head-to-tail knowledge transfer learning that enhances the performance of the model by transferring information of the head class to tail class in transfer learning. Among them, Feature Transfer Learning (FTL) performs tail-class feature augmentation based on variance information of the head-class. It is to address the issue of the variance in the tail class being much smaller than that of the head-class, thus resulting in the biased decision boundary. Online Feature Augmentation (OFA) extracts class-specific features and class-agnostic features based on the class activation map. It then augments the tail-class sample based on tail-class specific features and head-class agnostic features. Rare Class Sample Generator (RSG) estimates the feature center per class and strengthens the tail-class feature based on feature displacement between the head-class sample feature and the nearest intra-class feature center.

Representation Learning

Researches have recently been conducted actively using contrastive learning among metric learning to solve LTD issues. KCL proposes k-positive contrastive loss that can form a balanced feature space to reduce the LTD problem while increasing the generalization performance of the model. Parametric Contrastive Learning (PaCo) sets the class center as the weighted value of the model and adds a new class center that can conduct parametric learning to improve Supervised Contrastive Learning (SCL). Distributional Robustness Loss for Long-Tail Learning (DRO-LT) optimized distribution robustness based on Prototypical Contrastive Learning to make the model even more robust to data distribution shift.

Decoupled Learning

In the past, an end-to-end learning method for the feature extractor and downstream task model exhibited good performance in the balanced dataset, and most of the methodologies introduced were based on this. However, in the Decoupling Representation and Classifier for Long-Tailed Recognition paper recently introduced at ICLR 2020, it was proposed that if a dataset had an extreme LTD, much better performance could be obtained by dividing learning into two stages: (a) performing end-to-end learning using the existing method in stage 1 and (b) fixing the feature extractor and relearning the downstream task model in stage 2 as shown in Figure 5. Furthermore, during stage 1 representation learning and stage 2 classifier learning, various sampling strategies and models were used to evaluate performance. As a result, Instanced-Balanced Sampling (IBS) exhibited exceptionally good performance, and it was evident that performance could be surprisingly improved just by learning the classifier well.

Figure 5. Summary of Decoupling Representation and Classifier for Long-Tailed Recognition

Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-tailed Recognition

This study presents a new methodology for improving Out-Of-Distribution (OOD) detection task performance of a dataset with LTD. Most OOD detection methodologies proposed in the past were based on balanced distribution. When applying in LDT dataset, there were issues with a serious performance degradation as shown in Table 2. The main reason for this was the wrong prediction of the model between OOD samples and Tail-class In-Distribution (TID) samples. Therefore, this paper proposes Partial and Asymmetric Supervised Contrastive Learning (PASCL) that concentrates on increasing the prediction accuracy of OOD and TID samples.

Table 2. Comparison of Decreased Performance of LTD Dataset

Out-Of-Distribution Detection

OOD detection uses only in-distribution samples in training phase and detect OOD samples in inference phase which is unseen while learning. Deep Neural Network (DNN) reject OOD samples which is shown low-confident prediction result with high uncertainty. However, unlike our anticipations, existing DNN models often make erroneous high-confident predictions for new OOD samples as well. This is because TID is under-represented with a small number of samples and variance, thus making it difficult for the DNN model to find the OOD sample and decision boundary accurately. Figure 6 below shows that Head-class In-Distribution (HID) samples are well separated from OOD samples, while TID samples overlapped in large part with OOD samples.

Figure 6. Visualization of CIFAR10, CIFAR10-LT

The loss of existing OOD detection tasks is as follows. ${\mathcal{D}}_{in}$ and ${\mathcal{D}}_{out}$ correspond to the in-distribution and OOD training dataset, respectively. ${ℒ}_{in}$ is in-distribution classification loss and ${ℒ}_{out}$ is OOD detection loss, which learns to have the model show the low-confident result of the OOD sample. For example, in the Outlier Exposure (OE) task, ${ℒ}_{in}=\mathrm{XEntorpy}\left(\mathcal{x},\mathcal{y}\left(\mathcal{x}\right)\right)$ can be cross-entropy and Kl-divergence as shown in ${ℒ}_{in}=\mathrm{KL}\left(\mathcal{f}\left(\mathcal{x}\right)\parallel \mathcal{u}\right)$.

 

$ℒ={\mathbb{E}}_{\mathcal{x}~{\mathcal{D}}_{in}}\left[{ℒ}_{in}\left(\mathcal{x}\right)\right]+\lambda {\mathbb{E}}_{\mathcal{x}~{\mathcal{D}}_{out}}\left[{ℒ}_{out}\left(\mathcal{x}\right)\right]$

Method

This study claims the methodologies of the existing OOD detection and LTD recognition have conflicting objectives. The purpose of OOD detection is to make the model give under-confident results for rare samples while the purpose of LTD recognition is to train the model to have high prediction confidence of rare samples. Considering this, this paper aims at finding boundaries that can distinguish TIP samples and OOD samples better through PASCL, and separate the model into two branches using Auxiliary Branch Fine-Tuning (ABF), thereby making it possible to learn according to each objective.

Partial and Asymmetric Supervised Contrastive Learning (PASCL)

As shown in Figure 7, two aspects of Partiality and Asymmetry were improved from the existing SCL method to enhancing the prediction accuracy of OOD and TID samples.

Figure 7. Comparison of PASCL Proposed in this Paper Compared to Existing SCL

1. Partiality
As mentioned in the introduction, as HID samples are already well-separated from OOD samples, applying the same contrastive learning for all classes would be inappropriate. Therefore, contrastive learning is applied partially only to OOD and TID samples. TID samples within the same class are pulled to be located as close as possible in the feature space, and different TID and OOD samples are pushed so that they can be differentiated in the feature space.

2. Asymmetry
Unlike in-distribution samples, OOD samples are composed in various representation of each open world. Since it is unnecessary to pull OOD samples close to each other in the feature space, it is not pulled during training.

The main difference between PASCL and SCL are $\mathcal{I}$ and $\mathcal{A}\left(\mathcal{x}\right)$. When combining partiality and asymmetry, PASCL loss is as follows.

$\mathcal{L}={\mathbb{E}}_{\mathcal{x}~{\mathfrak{D}}_{in}}\left[{\mathcal{L}}_{in}\left(\mathcal{x}\right)\right]+{\lambda }_1{\mathbb{E}}_{\mathcal{x}~{\mathcal{D}}_{out}}\left[{\mathcal{L}}_{out}\left(\mathcal{x}\right)\right]+{\lambda }_2{\mathbb{E}}_{\mathcal{x}~\mathcal{I}}\left[{\mathcal{L}}_{\mathcal{c}}\left(\mathcal{x}\right)\right]$

$\mathcal{I}={\mathcal{D}}_{tail}, \mathcal{A}\left(\mathcal{x}\right)={\mathcal{D}}_{tail}\cup {\mathcal{D}}_{out}\backslash \left\{\mathcal{x}\right\}$

 

Auxiliary Branch Fine-Tuning (ABF)

Existing OOD detection SOTA methodologies include both in-distribution samples and OOD samples to conduct Batch Normalization (BN) using statistical values estimated at once. However, while this method can be appropriate for OOD detection, it is not the optimized one for in-distribution classification. Thus, this paper separately proposes the main branch that carries out OOD detection and an auxiliary branch that performs in-distribution classification. The learning process of the proposed method is composed of two stages: in the first stage, mixture-BN is used for both in-distribution and OOD samples to conduct learning to minimize the loss equation below. In the second stage, only in-distribution samples are used to fine-tune the auxiliary BN layer and classification layer of the model learned in the main branch. At this time, other convolutional layers are fixed and Logit-Adjustment (LA) cross-entropy loss is used to learn the model.

During inference, the new sample is placed in the main branch that performs OOD detection. If this sample is judged to be OOD, inference of the second branch is not performed. When judged to be in-distribution, in-distribution classification is performed through additional inference of the auxiliary branch.

The ABF loss equation is as follows.

${\mathbb{E}}_{\mathcal{x}~{\mathcal{D}}_{in}}\left[{\mathcal{L}}_{in}^{LA}\left(\mathcal{x}\right)\right]$

ABF Schematic Diagram

Experiment

In order to check the performance of the proposed method, this study uses a total of three long-tail image classification datasets including CIFAR 10-LT, CIFAR 100-LT, and ImageNet-LT. CIFAR 10-LT and CIFAR 100-LT used ResNet18 to verify the performance of the proposed method, and ImageNet-LT used ResNet50. In all three datasets, it was found that accuracy increased and the false positive rate reduced significantly. Furthermore, PASCL components such as asymmetry, partiality, and ABF ablation study results showed that the method including all components was the best.

Table 3. CIFAR10-LT Dataset Results

Table 4. Ablation Study Results of PASCL Components

Discussion

This paper proposed a new method that can improve OOD detection task performance with LTD datasets. Unlike existing SCL, PASCL was presented, which is a new methodology that performs contrastive learning to better classify TID and OOD samples. However, the assumption of HID samples was already well differentiated from OOD samples may be limited to real-world datasets. And another limitation is that the label information of in-distribution and OOD makes it hard to apply in unsupervised settings. It is necessary to think about a new way to differentiate TID samples and OOD samples in unsupervised settings to overcome this.

Conclusion

This paper examined the Long-Tail Distribution (LTD) and various methodologies to improve this. The LTD problem has been recognized as an important issue quite a long time ago. New methodologies are continuously being presented by integrating with various fields such as transfer learning, domain adaptation, and self-supervised learning, which are recently being researched actively. In addition, as it is an issue that appears commonly in various problems such as classification, regression, and out-of-detection, various approaches may exist. It is thus necessary to accurately identify the fundamental problems of LTD and continuously think about ways to improve the performance of downstream tasks.