[ICLR 2021] Part 2: Generative Model Trends at ICLR 2021

International Conference on Learning Representations(ICLR) 2021 is the 9th global conference on artificial intelligence and learning that primarily deals with the representation, learning, and optimization of deep learning models and their applications. Among a variety of academic papers available at ICLR 2021, which was held online from May 3 to 7, two researchers of LG AI Research who participated in the event would like to introduce to you two papers they have found particularly impressive. - Part 1: Reinforcement Learning Research to Solve Real Problems - Researcher Youngjoon Park - Part 2: Generative Model Trends at ICLR 2021 - Researcher Sanghyu Yoon

Online Conference using Metaverse
ICLR 2021 was held as a virtual conference for the second year due to COVID-19. As it was held online, it was structured to provide an informative and diverse experience on an easily accessible website. For the poster session, we used gather town, where instead of presenters, characters walked and took a look around the conference hall. When a character found a poster interesting, he/she could talk to the author through a video call. The conference fully incorporated “metaverse,” a space that has made headline in recent years, to offer participants a fun experience.



Generative Model at ICLR 2021
The fact that such keywords as generative learning, meta learning, and self-supervised learning appeared at the top at ICLR 2021 demonstrates that research on the methodology is widely conducted to address real-world problems. Since the Data Intelligence Lab of LG AI Research also deals with the current issues of various affiliates, departing from the standard supervised learning issues, a number of participants within the research lab have taken note of the generative model capable of solving a wide variety of issues as they have created plausible samples through the distribution estimation of data and detected the distribution alignment among domains and out-of-distribution. The generative model is past the period during which modifications based on Variational Autoencoder(VAE), Generative Adversarial Network(GAN) and Normalizing Flow(NF) pored out, and it is now creating meaningful generation performance such as the score-based model and energy-based model, accompanied by associated studies, as evident in the trend at ICLR 2021. In this regard, I would like to introduce you to two papers worthy to note.

Score-based Generative Modeling through Stochastic Differential Equation
In the score-based model, noises are added depending on the noise scale of the data and prior is created. At each step of this generative model, the score of perturbated data is estimated as the gradient of the log-likelihood and the score estimated through the reverse process is used to generate data from the prior. This paper was written by Yang Song, who proposed the famous Noise Conditional Score Network(NCSN)^[1] among the score-based generative models, and was selected as the best paper at ICLR 2021. It is a particularly meaningful paper due to the recent emergence of Score-based models that offer more stable learning and better performance than GAN.

Entire Process of the Score-based Generative Model through SDE

This paper considers the processing of adding noise to data as a stochastic process for continuous time t for the existing score-based model to obtain prior, and it relies on the solution of the stochastic differential equation(SDE), a differential equation including the stochastic process. The authors viewed the two existing models, score matching with Langevin dynamics(SMLD)^[1] and Denoising diffusion probabilistic modeling(DDPM)^[2] as discretization of SDE, and described each as Variance Exploding SDE(VE SDE) and Variance Preserving SDE(VP SDE) according to the perturbation kernel’s variance and integrated them into the SDE framework.

PC Sampler Algorithm

The authors also proposed a new sampling technique of reverse SDE to generate images from prior. This is a PC sampler with a Predictor-Corrector structure that predicts the previous time-step using a general purpose SDE solver and provides more detailed sampling with correction through the score-based Markov chain Monte Carlo(MCMC). Through improved sampling and network enhancement, the authors created the first high-fidelity image of 1024X1024 for the score-based model, and achieved SOTA in generation performance with CIFA-10.

VAEBM: A SYMBIOSIS BETWEEN VARIATIONAL AUTOENCODERS AND ENERGY-BASED MODELS
Energy-based model(EBM)^[3] is a generative model that not only learns the energy function that gives low energy to the input X within the data distribution and high energy to other inputs and maximizes the distribution in the part where data exists, but also reduces the distribution in the part where no data exists. If the input data acts as a positive sample that lowers the energy, then the output sampled from the boltzmann distribution defined by the energy function serves as a negative sample. Since sampling data from the boltzmann distribution is difficult, the MCMC-based sampling technique is used.

However, as MCMC-based sampling is accompanied by high computational cost, this paper suggested the VAEBM model that combined EBM to the VAE component for more efficient sampling. Compared to the score-based model, it records similar or slightly lower generation performance, but it displays a much faster sampling speed. In fact, the proposed model has largely caught up with the generation performance of GAN and the score-based model, displaying potential as EBM’s generative model.

 

Objective Function of VAEBM

The purpose of VAEBM is to maximize the data likelihood h_{ψ, θ}(x), and it is broken down into the loss function of VAE and EBM, respectively, as the above formula. L_EBM(x,ψ,θ) has VAE’s parameter, θ, involved, and thus, it is difficult to ensure joint optimization. This paper suggests two-stage’s learning algorithm to overcome computational difficulties.

Entire Process of VAEBM

In the first stage, VAE that acts as a generator is trained first. After training VAE, since θ is fixed, L_EBM(x,ψ,θ) optimizes ψ only, the parameter of the energy function, in the second stage. This two-stage algorithm does not only overcome the difficulties associated with optimization, but also reduces the steps required for update of ψ as pre-trained VAE already well resembles data distribution. Learning energy function may be regarded as VAE’s post-processing, which allows the depiction of an elaborate part as well as the primary part of data distribution, and therefore, it improves blur images, which has been the weakness of VAE, to demonstrate good generation performance. It also displayed better performance than the existing EBM model in the OOD detection.

Generative Model Research and Momentum
Thus far, the generative model research had the motivation of having the strengths of outlier detection, semi-supervised learning, and adversarial robustness through the estimation of data distribution. However, the recently proposed generative models, including the two papers of ICLR 2021, remain more focused on improving the generation performance, such as the quality of generated samples or log likelihood, than solving downstream problems. Compared to the rapid growth of the generative model, the fact that it does not produce better performance than the task-specialized model for discriminative problem that solves using the generative model has not been resolved yet. Going forward, further thought should put into how to transfer the knowledge obtained through relationship between input data of generative model to downstream problems. Since I am also concerned about improving the performance of discriminative problems using data with no labels for the tasks associated with our affiliates, I will continue to conduct related studies.