[NeurIPS 2023] Trends on Score-Based Generative Modeling

Recently, score-based generative modeling, or diffusion model research, has been gaining attention for its useful properties and excellent performance, with related research results being actively published. Many papers were presented on this topic at NeurIPS 2023. This post will look at the research trends in score-based generative modeling that were seen at NeurIPS 2023 and highlight some impressive research papers.

1. Diffusion Model Architecture

The loss of the diffusion model is a seemingly simple L2-loss, but there are many different weights and hyperparameters inside. NVIDIA’s EDM^[1], which provides an overview of these hyperparameter settings, was named an Outstanding Paper at NeurIPS 2022. At NeurIPS 2023, Google DeepMind’s “Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation”^[2] was selected as an Oral Presentation. This paper discusses weighting settings that combine training loss in diffusion models with different noise levels.

If the weighting is 1, this corresponds to the ELBO term, but most diffusion models use different weights for different purposes (perceptual quality).

 

Figure 1. Different weighting^[2]used by current diffusion models

 

This study proves that monotonic weighting can be represented by ELBO when data is augmented with a Gaussian Perturbation and designs a monotonic form of weighting for the EDM objective. The authors applied this weighting to the ImageNet dataset at various resolutions and achieved consistent and significant performance gains over traditional scheduling.

In addition to the above papers, two papers^[3,4] that took a performance approach to the U-Net architecture over the base architecture of the diffusion model were also impressive.

 

▶ C. Williams et al. A Unified Framework for U-Net Design and Analysis, NeurIPS 2023^[3] (Link)

An indispensable base structure for an image-based diffusion model is the U-Net architecture. This study reduces the number of parameters to be learned by configuring the encoder part of the existing residual U-Net with a basis function such as a wavelet. This configuration creates a general U-Net structure that allows for the use of multiple bases and can be applied to geometries such as spheres. Through experiments, this paper shows the superior performance of the designed multiresolution U-Net for the same hyperparameter size.

▶ Y. Li et al., SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Seconds, NeurIPS 2023^[4] (Link)

You have probably spent minutes waiting on an Internet server to create the prompts you want with a text-to-image diffusion model. While the text-to-image model’s sampling speed is said to have increased, it is still limited by long sampling times of several minutes. This paper presents a method for speeding up the sampling of the U-Net decoder portion of a large text-to-image model through data distillation. It can also be combined with various training strategies to produce effective and fast text-to-image models that can be processed on-device.

 

2. Text-to-Image Diffusion Model : Downstream Task

Since the latent diffusion model (LDM) and stable diffusion^[5] were introduced at CVPR 2022, much research has been conducted utilizing the text-to-image diffusion model. At NeurIPS 2023, papers applying not only existing generations but also various downstream tasks were introduced. Among these, the paper I would like to introduce is “Text-to-Image Diffusion Models are Zero-Shot Classifiers”^[6] from Google DeepMind.

 

Figure 2. Zero-Shot Classification Process^[6]

 

In this paper, prompts related to each class are given as conditions and each result is generated as guidance. Then, zero-shot classification is performed by providing the difference from the existing model as a score and summing it up at several timesteps, using the class with the minimum difference as the output. This method showed better performance than the zero-shot classification of models, such as existing CLIP, and showed slightly more robust results even with texture bias, which is prone to classifier overfitting. Although it is based on a diffusion model and requires a lot of evaluation time, it looks promising.

In addition to the above paper, papers on topics like goal-conditioned RL^[7]and point-cloud completion^[8] were also interesting studies that confirmed the potential of the text-to-image diffusion model for several downstream tasks. Some studies treated multiple tasks and condition modalities with one diffusion model. Uni-ControlNet^[9]made local conditions such as pose and segmentation maps compatible with stable diffusion through local control adaptors. A prompting condition was later added to the text token via a global control adaptor. The model performed well against the existing baseline, even when conditioned to different modalities.

▶ J. Gao et al., Can Pre-trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?, NeurIPS 2023^[7] (Link)

The application of score-based generative models to reinforcement learning (RL) has grown significantly since a paper presented at ICML in 2022^[10]. However, most papers modeled (state, action) trajectories^[11], policies^[12], or hyperparameters of these distributions^[13], and used models with relatively simple architecture. This study used a pre-trained latent diffusion model and language instruction to generate the goal state to be reached in the reinforcement learning task and augmented it with positive samples to the reinforcement learning backbone algorithm. It is an impressive study that extends the generative possibilities of the text-to-image diffusion model to RL.

LG AI Research’s Advanced Machine Learning (AML) lab is not only conducting theoretical research on various generative models, including diffusion models, but also extensively researching the possibility of applying the models to various downstream tasks. We will continue to analyze global research trends and create practical technological advances and outcomes.

▶ Want to learn more about the NeurIPS 2023 Research Blog series? (Link)