[ICML 2022] Part 2: Self-supervised learning

ICML 2022 (International Conference on Machine Learning 2022) is a large-scale international academic conference marking its 39th year this year. It has a citation index of 20% and an impact factor of 6.99. Participated by more than 70,000 people every year, it is deemed to be one of the most influential AI academic conferences. LG AI Research also took part in the ICML 2022 held both online and offline in Baltimore, Maryland from July 17 to 23. Three researchers from LG AI Research have selected meaningful studies that were shared at the conference. - Part 1: Long-Tail Distribution Learning – Suhee Yoon, Applied AI Research Lab - Part 2: Self-supervised learning – Hyemin Jung, Applied AI Research Lab - Part 3: Neural Combinatorial Optimization studies – Hanseul Jeong, Applied AI Research Lab

Introduction

Self-supervised learning, which is one of the unsupervised learning methods, learns meaningful data representation from a dataset without labels. Self-supervised learning is divided into two directions. The first is the “self-prediction” method which looks at parts from a data sample to predict other parts to learn representation. The second is “contrastive learning.” It learns embedding space that brings similar samples closer and different samples further away from each other. Many studies are being carried out in the two directions above, and multiple papers related to self-supervised learning were presented at this year’s 2022 ICML. In this blog, two papers that were especially impressive among the self-supervised learning papers presented at the 2022 ICML will be explained.

Adversarial Masking for Self-Supervised Learning^[1]

Motivation
Masked image modeling is a way of learning representation, which masks part of an image data and then restores it. This was conceived from language modeling methods such as BERT. In the case of language, words with a single semantic meaning are masked like the figure 1 below, and therefore, when the model restores a masked word, it uses nearby context information.

In the case of images, however, parts of pixels are randomly masked and restored as shown in the figure 2 below.

In this case, it does not use nearby context information, but uses local correlation for easy restoration. This method has the problem that representation containing semantic information of an image will not be properly learned. Therefore, in order to learn representation that properly contains semantic information of an image, the part that is being masked becomes very important.

This paper proposes a self-supervised learning framework to learn which part to mask. The name of the proposed method is “ADIOS.” The results of masking with the proposed method are as follows, which will be explained in detail.

Method
The general structure is composed of the inference model and occlusion model as shown in the figure 4.

The occlusion model (M), the left side of the figure 4, is a model that generates the original image-sized mask. The inference model (I), the right side of the figure 4, is a model that receives input of the occluded and original images to calculate the representation of each. The inference model only used the encoder part. This is because there were recent research results claiming that pixel level reconstruction using the generative model would actually waste modeling capacity and exhibit subpar performance.

The two models learn adversarially as follows.

When training the model with the above method, the occlusion model will maximize the distance between the masked version and the representations of the original, while the inference model will minimize the distance between these two. Let’s now take a closer look at each model.

<Inference model>

The expected role of the encoder is to use the observed context to restore the missing parts when there is a masked image, and calculate the representation of the entered image. In other words, the representation from entering the masked image in the encoder must be the same as the representation from entering the original image in the encoder. The below objective function is proposed for this. (D is the metric for calculating distance.)

When the above equation is learned, the encoder reasons the masking part of the masked image to minimize the distance between the two representations.

The inference model structure can use the models that use augmented image views (SimCLR, BYOL, and SimSiam) among existing self-supervised models. The new models all use two views, and the common method for applying ADIOS in these models is to apply the mask from the occlusion model and enter it.

This is the structure and objective function applying ADIOS proposed by SimCLR.

This is the structure and objective function applying ADIOS proposed by SimSiam

This is the structure and objective function applying ADIOS proposed by BYOL.

<Occlusion model>

The occlusion model can generate one or multiple masks. When each mask masks other components of the image, the inference model reasons the masked parts to learn the relations. The structure of the occlusion model uses the U-Net structure. Using N 1*1 convolutions in the feature map with the original size in U-Net, the final N number of masks will be generated. It can binarize the mask value as 0 and 1 or directly use the real-valued masks.

<General Structure>

The structure of the model that creates N number of masks is as follows.

The total loss equation is the average of the losses that used N number of masks as follows.

But when learning the above objective function, all of the mask values from the occlusion model can be either 0 or 1. In order to prevent this, the paper suggested additionally using the below sparsity penalty.

By additionally minimizing the above sparsity penalty, it prevents all of the mask values from becoming 0 or 1.

The final loss equation applying the above penalty is as follows.

In order to calculate the above equation, N number of encoder forward passes are needed, and this is computationally expensive. Therefore, this paper additionally proposes a lightweight version as follows.

Experiment
The results of learning representation with self-supervised learning applied with the proposed method and classification and transfer learning using this are as Table 1.

Self-Supervised Representation Learning via Latent Graph Prediction^[2]

Motivation
The method for applying the self-supervised learning in the graph is mainly divided into contrastive method and predictive method. The predictive method reconstructs the parts masked in the masked input graph, such as node reconstruction and connectivity reconstruction. This study proposed self-supervised learning based on the predictive method.

Method
The goal of the problem is to suggest the pre-text learning task to calculate meaningful node-level and graph-level representation when graph information such as adjacency matrix and node features are entered in the encoder.

In order to understand the pre-text learning task proposed in this paper, the concept of latent data should be explained first. In the given observed data instance x, there is latent data $x_I$ that responds to it. The semantic information of the observed data instance x is determined by the latent data. Generally, observed data is the noisy version and latent data is the clean version. When applying this concept to the graph data, the observed graph data G=(A,X) is a noisy graph generated from latent graph $G_I$=(A,F). The two have the same node set and edge set, and node feature matrices X and F have the same dimensionality. Since latent data determines the semantic meaning of the observed data, pre-text learning task that uses latent data as the supervision signal was prescribed. While predicting the latent data, representation of meaningful node level and graph level can be learned.

The objective function of the latent graph prediction task that applies this is as follows.

<Node-level Representation>

The upper bound for calculating the representation of each node is as follows.

The upper bound of the objective function that predicts the node feature F of the latent graph is made up of two terms. The first term is the reconstruction term which makes the representation information of the node from the encoder informative. The second term is the invariance regularization term which reconstructs the input feature of a node without giving the features of the corresponding node.

The entire process for calculating the above objective function is as figure 10.

Experiment
The proposed method was applied to train the encoder that calculates the node-level representation and graph-level representation to proceed with the experiment. The node-level representation from the learned encoder was fixed and after learning the linear classifier, the evaluated results are as Table 3.