ICML 2022 (International Conference on Machine Learning 2022) is a large-scale international academic conference marking its 39th year this year. It has a citation index of 20% and an impact factor of 6.99. Participated by more than 70,000 people every year, it is deemed to be one of the most influential AI academic conferences. LG AI Research also took part in the ICML 2022 held both online and offline in Baltimore, Maryland from July 17 to 23. Three researchers from LG AI Research have selected meaningful studies that were shared at the conference.

- Part 1: Long-Tail Distribution Learning – Suhee Yoon, Applied AI Research Lab
- Part 2: Self-supervised learning – Hyemin Jung, Applied AI Research Lab
- Part 3: Neural Combinatorial Optimization studies – Hanseul Jeong, Applied AI Research Lab

Introduction

Self-supervised learning, which is one of the unsupervised learning methods, learns meaningful data representation from a dataset without labels. Self-supervised learning is divided into two directions. The first is the “self-prediction” method which looks at parts from a data sample to predict other parts to learn representation. The second is “contrastive learning.” It learns embedding space that brings similar samples closer and different samples further away from each other. Many studies are being carried out in the two directions above, and multiple papers related to self-supervised learning were presented at this year’s 2022 ICML. In this blog, two papers that were especially impressive among the self-supervised learning papers presented at the 2022 ICML will be explained.

Adversarial Masking for Self-Supervised Learning^[1]

Motivation
Masked image modeling is a way of learning representation, which masks part of an image data and then restores it. This was conceived from language modeling methods such as BERT. In the case of language, words with a single semantic meaning are masked like the figure 1 below, and therefore, when the model restores a masked word, it uses nearby context information.

Figure 1

In the case of images, however, parts of pixels are randomly masked and restored as shown in the figure 2 below.

Figure 2

In this case, it does not use nearby context information, but uses local correlation for easy restoration. This method has the problem that representation containing semantic information of an image will not be properly learned. Therefore, in order to learn representation that properly contains semantic information of an image, the part that is being masked becomes very important.

This paper proposes a self-supervised learning framework to learn which part to mask. The name of the proposed method is “ADIOS.” The results of masking with the proposed method are as follows, which will be explained in detail.

Figure 3

Method
The general structure is composed of the inference model and occlusion model as shown in the figure 4.

Figure 4. ADIOS Architecture

The occlusion model (M), the left side of the figure 4, is a model that generates the original image-sized mask. The inference model (I), the right side of the figure 4, is a model that receives input of the occluded and original images to calculate the representation of each. The inference model only used the encoder part. This is because there were recent research results claiming that pixel level reconstruction using the generative model would actually waste modeling capacity and exhibit subpar performance.

The two models learn adversarially as follows.

When training the model with the above method, the occlusion model will maximize the distance between the masked version and the representations of the original, while the inference model will minimize the distance between these two. Let’s now take a closer look at each model.

<Inference model>

The expected role of the encoder is to use the observed context to restore the missing parts when there is a masked image, and calculate the representation of the entered image. In other words, the representation from entering the masked image in the encoder must be the same as the representation from entering the original image in the encoder. The below objective function is proposed for this. (D is the metric for calculating distance.)

When the above equation is learned, the encoder reasons the masking part of the masked image to minimize the distance between the two representations.

The inference model structure can use the models that use augmented image views (SimCLR, BYOL, and SimSiam) among existing self-supervised models. The new models all use two views, and the common method for applying ADIOS in these models is to apply the mask from the occlusion model and enter it.

This is the structure and objective function applying ADIOS proposed by SimCLR.

Figure 5

This is the structure and objective function applying ADIOS proposed by SimSiam

Figure 6

This is the structure and objective function applying ADIOS proposed by BYOL.

Figure 7

<Occlusion model>

The occlusion model can generate one or multiple masks. When each mask masks other components of the image, the inference model reasons the masked parts to learn the relations. The structure of the occlusion model uses the U-Net structure. Using N 1*1 convolutions in the feature map with the original size in U-Net, the final N number of masks will be generated. It can binarize the mask value as 0 and 1 or directly use the real-valued masks.

<General Structure>

The structure of the model that creates N number of masks is as follows.

Figure 8. ADIOS, N>1

The total loss equation is the average of the losses that used N number of masks as follows.

But when learning the above objective function, all of the mask values from the occlusion model can be either 0 or 1. In order to prevent this, the paper suggested additionally using the below sparsity penalty.

By additionally minimizing the above sparsity penalty, it prevents all of the mask values from becoming 0 or 1.

The final loss equation applying the above penalty is as follows.

In order to calculate the above equation, N number of encoder forward passes are needed, and this is computationally expensive. Therefore, this paper additionally proposes a lightweight version as follows.

The lightweight version randomly samples one from the N number of generated masks to apply in the input image. We will explain the results experimented with the method proposed in the study.

Experiment
The results of learning representation with self-supervised learning applied with the proposed method and classification and transfer learning using this are as Table 1.

Table 1

When applying ADIOS in the classification task, performance was improved for all.

After conducting self-supervised learning with ImageNet100-S, transfer learning was carried out for four different datasets. As a result, performance improved for all, and the results are as Table 2.

Table 2

In addition, the mask generated from the occlusion model was analyzed. As shown in the figure 9, it was generated well in the part where the mask was semantically-meaningful and it was also found that mask was granted to different levels by dataset.

Figure 9

Self-Supervised Representation Learning via Latent Graph Prediction^[2]

Motivation
The method for applying the self-supervised learning in the graph is mainly divided into contrastive method and predictive method. The predictive method reconstructs the parts masked in the masked input graph, such as node reconstruction and connectivity reconstruction. This study proposed self-supervised learning based on the predictive method.

Method
The goal of the problem is to suggest the pre-text learning task to calculate meaningful node-level and graph-level representation when graph information such as adjacency matrix and node features are entered in the encoder.

In order to understand the pre-text learning task proposed in this paper, the concept of latent data should be explained first. In the given observed data instance x, there is latent data $x_{I}$ that responds to it. The semantic information of the observed data instance x is determined by the latent data. Generally, observed data is the noisy version and latent data is the clean version. When applying this concept to the graph data, the observed graph data G=(A,X) is a noisy graph generated from latent graph $G_{I}$ =(A,F). The two have the same node set and edge set, and node feature matrices X and F have the same dimensionality. Since latent data determines the semantic meaning of the observed data, pre-text learning task that uses latent data as the supervision signal was prescribed. While predicting the latent data, representation of meaningful node level and graph level can be learned.

The objective function of the latent graph prediction task that applies this is as follows.

However, as observed graph data G=(A,X) is given while latent graph $G_{I}$ =(A,F) information is unknown, the above objective function cannot be calculated right away. In order to solve this problem, an upper bound that substitutes the above objective function was proposed in this study.

<Node-level Representation>

The upper bound for calculating the representation of each node is as follows.

The upper bound of the objective function that predicts the node feature F of the latent graph is made up of two terms. The first term is the reconstruction term which makes the representation information of the node from the encoder informative. The second term is the invariance regularization term which reconstructs the input feature of a node without giving the features of the corresponding node.

The entire process for calculating the above objective function is as figure 10.

Figure 10

When G=(A,X) is given, graph G’ that masked parts of the sub-set node are created here. The encoder is entered in each of the graphs to calculate the embedding of the nodes. (H and H' calculation) H is entered in the decoder made up of fully connected layers to restore the features of all nodes, and this value is used to calculate the reconstruction term. In order to calculate the invariance regularization term, the difference in the features of the masked note among H and H' are calculated.

<Graph-level Representation>

The upper bound for calculating the representation of the graph is as follows.

The entire process for calculating the above objective function is as figure 11.

Figure 11

The method for calculating node-level representation is different from just calculating the invariance regularization term. Embedding z,z’ of the graph generated is calculated by passing H, H’ in the read out function. The difference in the representation of each graph found is then calculated.

Experiment
The proposed method was applied to train the encoder that calculates the node-level representation and graph-level representation to proceed with the experiment. The node-level representation from the learned encoder was fixed and after learning the linear classifier, the evaluated results are as Table 3.

Table 3

The graph-level representation from the learned encoder was fixed and after learning the linear classifier, the evaluated results are as Table 4.

Table 4

Results of both experiments showed that the performance of the proposed method was better than the method that applied the self-supervised learning in the existing graph.

Conclusion

To summarize the two papers, “Adversarial Masking for Self-Supervised Learning” proposed a method for generating a mask that helps learn good representation from image data, and “Self-Supervised Representation Learning via Latent Graph Prediction” suggested a predictive self-supervised learning method from the graph domain, thus setting SOTA records.

Apart from the two papers above, interesting self-supervised learning studies such as methods for applying meta-learning concepts and methods for applying in various domains such as image and language were presented at the 2022 ICML conference.