[NeurIPS 2022] Conference Review and Latest Research Trends

NeurIPS 2022 (Neural Information Processing Systems), considered as one of the world’s top AI conferences, was held in New Orleans, USA from November 28 to December 9, 2022, in a hybrid format of in-person as well as virtual programs. Many researchers from LG AI Research have attended and conducted various activities at the conference, such as presenting new publications, exhibiting technology demonstrations, and operating a corporate booth with LG affiliate companies. During the conference, we were able to check out latest global research trends and gather further insight thanks to numerous papers presented at NeurIPS 2022. In this blog post, we provide reviews of several noteworthy papers on Generative Modeling, Large Language Models, and Reinforcement Learning, written by Sungik Choi, Dasol Hwang, and Geonhyeong Kim of the Advanced ML Lab (AML).

[AML Sungik Choi] Score-based Generative Modeling

A topic that has lately been in the spotlight in the generative model field is the score-based generative model (SGM). From its reemergence at NeurIPS 2019^[1], SGM has already established itself as the standard in image synthesis^[2], text-to-image modeling^[3], etc. Here is a summary of the trends in the SGM research presented at NeurIPS 2022.

Due to the multiple sampling steps that SGM requires, SGM suffers from slow generation speed compared to its competitors. So, numerous studies^[4,5,6] to increase SGM generation speed were introduced at NeurIPS. Of these, we will introduce two studies that demonstrated how to speed up SGM sampling without altering the pre-trained network.

First, DPM-solver^[5] is a method in which the probability flow ordinary differential equation corresponding to the SGM’s reverse process forms a semi-linear structure. As a result, a sampling technique that reduces approximation error by approximating just one non-linear term instead of the entire ODE was designed.

GENIE^[6] proposed a high-order solver that would better reflect the probability flow ODE based on truncated Taylor’s method

Figure 1. Motivation of High-order Solver^[6]

In addition, by adding a small neural network on top of the existing model and approximating it to match the gradient of the training data, the amount of additional computation required to obtain the gradient was reduced at the inference phase.

Figure 2. Distillation network of GENIE^[6]

The methods proposed above showed performance improvement in various image domains. In particular, in the CIFAR-10 dataset, GENIE significantly reduced the number of function evaluations compared to the pre-existing ddpm, which is often used to evaluate the sampling speed of diffusion models. It also showed stable performance compared to other methods, even when the number of function evaluations was around 10, given that original DDPM requires thousands of function evaluations.

Figure 3. FID comparison function evaluation on the CIFAR-10 dataset of GENIE^[6]

Studies that define SGM in various spaces other than the existing image domain also played a large role at NeurIPS 2022. The performance of image generation is enhanced by INDM^[7], which defines diffusion in latent spaced induced from a flow network. The Riemannian SDE^[8] has been shown to better reflect the manifold by defining the SDE in the Riemannian geometry as opposed to the general Euclidean space.

Figure 4. Riemannian neural field over Riemannian Neural SDE^[8]

Lastly, SGM has been successfully applied for various visual downstream tasks. In particular, studies that use unconditional SGM for downstream tasks in a plug-and-play-format without additional training have also appeared. For example, problems like super-resolution and inpainting can be formulated as linear inverse problems. It was shown that DDRM^[9] formulates the noisy linear inverse problem through sampling in the spectral space obtained through singular value decomposition and that efficient sampling is possible with only the unconditional diffusion model without additional training. By using the output denoised through Tweedie’s formula in the sampling path rather than the noised data, MCG^[10] demonstrated state-of-the-art performance in various noise-less linear inverse problems.

Figure 5. Manifold Constraint gradient for improved sampling under inverse constraint^[10]

The AML Lab at LG AI Research is focused on studying SGM-based generative models, conducting extensive research on text-to-image modeling and image generation as well as fundamental research on neural SDE and its application to audio data. We are further planning to extend these results to large-scale generative models.

[AML Dasol Hwang] Large Language Models

The performance of large-scale models trained on enormous amounts of data has been remarkable across a wide range of domains, including text, image, and audio. Despite their impressive results, the training of such models requires significant resources, including computing pawer and budget, which increase proportionally with the size of model. As available resources are usually limited, extensive research is being conducted on how to effectively train large-scale models using given resources. At NeurIPS 2022, many studies related to large-scale models were presented, and among them, these two studies that received attention will be introduced.

Training Compute-Optimal Large Language Models^[11]

This paper published by DeepMind details various experiments and analyses on the correlation between the size of the model and the amount of data that can produce the optimal performance of a large-scale language model using limited resources. In particular, it presents a way to find parameters and training tokens that can produce the best performance given the model’s floating-point operations (FLOPs). In addition, it was discovered that the number of learning tokens in existing large-scale language models is small in comparison to the increase in model size, (see Figure 6) and it could be seen that a model trained with more data in a smaller model outperformed existing models.

Figure 6. Optimal values of the number of learning tokens and the number of parameters for given training FLOPs^[11]

Figure 7. FLOPs-Parameters plot ^[11]

This paper presents Chinchilla (70B), a model that is approximately 4 times smaller than Gopher (280B), an existing large-scale language model based on the same FLOPs. (see Figure 7) The authors discovered that the number of training tokens was small for the existing large-scale language models compared to the increase in model size. It was confirmed that Chinchilla, which used approximately 4 times more learning tokens with 1/4 times the model size of Gopher, outperformed the Gopher model when applying the same FLOPs standards. What’s more, the 70B Chinchilla presented in this paper showed superior performance than the 280B Gopher, the 175B GPT-3, and the 530B Megatron-Turing NLG. This means that because Chinchilla is smaller than existing large models, so fewer resources are used for fine-tuning and inference.

Flamingo: a Visual Language Model for Few-Shot Learning^[12]

In existing multi-modal studies, pre-training with a large amount of supervised data had been a typical paradigm, followed by fine-tuning for the primary task of interest. However, even with fine-tuning, massive amounts of data with tens of thousands of annotations are occasionally required, and each task frequently requires hyperparameter adjustments, leading to resource dependence. Accordingly, studies on zero-shot adaptation that can be performed in unseen tasks without fine-tuning have recently been introduced in the multimodal field.

Figure 8. Model structure of Flamingo^[12]

Flamingo is a model capable of prompt-based in-context zero/few-shot learning, and starting with a pretrained vision-only, language-only model, it learns a Multi-modal Vision-Language Model. A characteristic of Flamingo is that it freezes each unimodal model and learns only the layer that acts as a bridge between the two modalities. This saves computational resources required to train unimodal models from scratch and makes it possible to utilize the abilities of each unimodal model. In addition, since it is possible to handle sequences of arbitrarily interleaved visual and textual data, Flamingo can be trained on large-scale multimodal web corpora. By linking unimodal models that only understood each modality, better performance is shown compared to fine-tuned models on multiple benchmarks through a small number of examples.

[AML Geonhyeong Kim] Multi-modal Behavioral Cloning: Behavior Transformers^[13]

Since the recent remarkable achievements of transformer-based algorithms in the NLP field, researchers have begun to propose studies to utilize them in reinforcement learning, like decision transformers^[14] and trajectory transformers^[15]. Since then, based on these papers, the number of studies related to reinforcement learning using the transformer structure is gradually increasing^[16,17]. Among the various transformer-based studies^[18-20] presented at NeurIPS 2022, Behavior Transformer (BeT)^[13], a study using transformers in imitation learning, will be introduced.

How to learn policies from pre-collected data is a fundamental issue that has been consistently necessary for various domains, including autonomous driving, robot control, and more. Methods for learning behavior from pre-collected data are largely divided into methods using offline reinforcement learning and methods using imitation learning, in particular behavioral cloning. In real life, specific rewards are not given in many cases, and behavioral cloning is widely used in such cases where direct rewards are not given. Generally, behavioral cloning is learned through supervised learning that predicts an action for a given state from pre-collected data, and with enough data, it demonstrates excellent performance. However, many of the latest behavioral cloning algorithms assume that data is collected from a unimodal expert policy. However, when generating a sentence with a specific meaning or solving a given task, people generate many different types of sentences or solve given tasks in many different ways. Because of this, it is common for data collected from real people to be multi-modal

Figure 9. A comparison of policy learned with an existing behavioral cloning algorithm and with BeT proposed in this paper for given multi-modal data^[13]

BeT, a new methodology for learning from this multi-modal data, is composed based on three main factors:

1. Utilizing the context-based multi-token prediction capabilities of transformer-based models, it predicts multi-modal actions.

2. Transformer-based models are appropriate for discrete classification, and so k-means clustering is utilized to cluster continuous multi-modal behavior into k number of clusters.

3. The residual action corrector is learned so that continuous action can be generated in BeT.

To this end, the authors compose BeT by utilizing k-means clustering and a transformer decoder called MinGPT^[21]. The composed BeT learns over two steps:

1. It learns to cluster given continuous actions into k number of clusters using a k-means algorithm. Here, the central point of each learned cluster is called the “action center.” This makes an encoder that generates action centers and offsets from continuous actions, as well as a decoder that generates continuous actions from action centers and offsets. (See A in Figure 10)

2. Sub-paths are extracted from paths in the dataset, and these observations are entered into MinGPT. At this time, the categorical distribution and action offset generated by MinGPT use the action center and offset generated by the k-means encoder as the ground truth to compose the loss function. (See B in Figure 10)

Afterward, when utilizing it for actual rollout, these observations are entered into MinGPT, and the generated discrete actions and action offsets are put into a k-means decoder to finally generate continuous actions to be actually used. (See C in Figure 10)

Figure 10. BeT architecture structure^[13]

Lastly, the authors conducted experiments in three domains: self-driving CARLA^[22] based on visual observation, two robot control domain Block Push^[23], and Franka Kitchen^[24], a variety of tasks in a kitchen environment. As shown in Figure 11, BeT showed superior success rates compared to conventional algorithms when it came to most tasks.

Figure 11. Performance comparison of BeT and existing algorithms.
In CARLA, the percentage of reaching the target point, in Block Push, the percentage of successfully reaching each block and the percentage of pushing them to the desired location,
and in Kitchen, the percentage of the completed n number of tasks in 280 steps^[13]

This paper proposed a BeT algorithm that is able to handle multi-modal data using a transformer structure in order to improve existing behavioral replication algorithms that deal only with unimodal expert data. If this is developed further, it is expected that multi-modal policy learning closer to that of real people will be possible.

Conclusion

As NeurIPS is a general AI conference targeted towards a wide audience, papers presented at NeurIPS 2022 have again demonstrated an extremely broad variety of topics. Among them, we reviewed several major papers in the fields of Generative Modeling, Large Language Models, and Reinforcement Learning, which are topics of great interest at LG AI Research.

At this conference, LG AI Research presented excellent research results through a total of 12 publications, seven at the main conference and five at the workshops. LG AI Research also organized an event for networking with researchers in academia, inviting students majoring in AI from schools around the world to an LG AI Day dinner. After attending so many presentations at NeurIPS 2022, it became quite clear that the scope of AI applications is still expanding. In light of such growth, the AML Lab at LG AI Research will continuously aim towards advancing fundamental research that can be applied to various domains.