[CVPR 2022] LG AI Research joins CVPR 2022, the Mecca of Vision Research

From June 21st to June 24th, CVPR 2022 (Computer Vision and Pattern Recognition Conference) was held for four days. It was the first time in two years that the conference was held face-to-face in New Orleans under the scorching heat of the sun. Vision researchers around the world gathered and shared their research with enthusiasm. In order to be at the forefront of vision research, the “eye to see the world,” LG AI Research also set up a corporate booth with LG affiliates.

In this post, we will discuss CVPR 2022 with researchers Taehoon Kim, Bumsoo Kim, and Sungmin Cha of Vision Lab (VL) and Fundamental Research Lab (FRL).

[VL Taehoon Kim] Multimodal Representation: Ultimately, everything is simply data composed of 0s and 1s

The majority of vision research, including Visual Recognition, Image Synthesis, and Style Transfer, has centered on extracting meaningful information from pixel-based images or generating/transforming images. However, just as the human eye cannot comprehend everything, the need to solve more complex problems by combining images, texts and sounds have emerged in AI research, together with the growing importance of Multimodal Representation Learning. At CVPR 2022, the most recent multimodal research that combines Vision with other fields was presented, among which we will introduce a study that was chosen as an outstanding oral presentation.

(Figure 1) Rowan Zellers, the first author, presented a poster presentation for MERLOT Reserve^[1]

MERLOT Reserve^[1] is a study that collects information contained in vision, language, and sound included in the video and applies it to a variety of video analysis problems, including Video Common Sense (VCR) and Video Question & Answering (VQA). In contrast to previous studies that employed separate vision, language, and sound analysis models, this study employs a single transformer model in the pre-learning phase to simultaneously learn images, text subtitles and sounds for each frame of the video.

(Figure 2) Overview of the MERLOT Reserve model (left) and a description of the training procedure (right).

MERLOT Reserve learns comprehensively the relationship between the video's image, text, and audio information.

According to the author Rowan Zellers, MERLOT Reserve, which was trained using images, text subtitles, and sounds from 20 million videos, was able to achieve superior performance in a variety of tasks when compared to the previous model, which was trained using only images and text subtitles. Zellers says that adding sound information to learning played a crucial role in this process.

(Figure 3) Taehoon Kim delivering an oral presentation on L-Verse ^[2] (Top),

Seunghwan Kim, Leader of Vision Lab, with Taehoon Kim, preparing a poster presentation (Bottom)

Several studies that combine images with other information, such as language or sound, were presented at CVPR 2022, including Vision + Language, Vision + Sound, and MERLOT Reserve. Oral and poster presentations on the subject of L-Verse^[2] were also delivered, centering on the topic of bidirectional generation between image and text. In the previous post (link), you can view the most recent developments in the Large Vision Language Model (LVLM), including L-Verse. In addition, you can view the CVPR presentation video introducing L-Verse on LG AI Research's official YouTube (link) channel.

One pixel, the smallest unit that expresses image data in the digital world, is comprised of red, green, and blue (RGB) color units, with each color occupying one byte to make a total of three bytes. In addition, the smallest unit of text is one byte (ASCII standard). A byte is subdivided into eight bits, each of which can only represent 0 or 1. Ultimately, all information in the digital world consists of 0s and 1s, and Multimodal Representation Learning is centered on how this is comprehensively analyzed. The research presented at CVPR 2022 is anticipated to advance in the future.

[VL Bumsoo Kim] Neural Radiance Field (NeRF): The form of an object derived from a single photograph

Neural Radiance Field (NeRF)^[3], which has made a significant impact in the field of View Synthesis since ECCV 2020, was the most discussed topic at this year's CVPR. NeRF is the most recent method for View Synthesis that generates the appearance of an object when viewed from a previously unseen perspective by combining images captured from multiple angles. There are a total of 29 papers with NeRF in their titles that were selected for CVPR 2022, and eight of them were selected as excellent presentation papers in the top four percent.

View Synthesis was until recently one of the greatest obstacles in computer vision. This is due to the fact that, unlike general 2D images, the shape and color arrangement of actual 3D objects visible to our eyes varies based on the location of the “light source.” In this post, we will introduce Pix2NeRF^[4], which overcomes the limitations of the current NeRF, among other studies selected for CVPR 2022.

(Figure 4) Once learning is complete, Pix2NeRF can generate multiple images from a single image.

Even in the inference phase using the trained model, the previous NeRF required multiple images of a single object. However, Pix2NeRF generates a multi-view image from a single image using a generative adversarial network (GAN), enabling single-shot NeRF as opposed to few-shot. In order to secure data collected from multiple perspectives on a single object, there are spatial and financial limitations; however, Pix2NeRF presents a new direction for NeRF research and demonstrates the capability of overcoming these limitations at the inference stage.

NeRF-related research can be applied to a variety of fields, including 3D avatar generation, object-centric novel view synthesis, and 3D-aware super-resolution, thereby attracting interest from a variety of industrial fields including AI human, VR, and AR. In addition to Pix2NeRF, a number of the most recent NeRF research outcomes were presented at CVPR 2022. The most recent NeRF research trends revealed at CVPR 2022 can be viewed in detail in a separate post (link).

[FRL Sungmin Cha] Continual Learning: First Steps Towards Sustainable Model Learning

In a variety of domains, such as Image Classification, Object Detection, and Semantic Segmentation, models employing artificial neural networks exhibit human-level or even superior performance. This result, however, is limited to a model that performs a single task, and continual learning, which continuously learns new tasks, still demonstrates a significant gap with humans. For example, even when humans learn new task skills in the context of learning a sequence of tasks, they do not forget how to perform previously learned tasks; however, in the case of artificial neural networks, there is catastrophic forgetting, wherein the majority of information regarding previously learned tasks is lost.

Consequently, the field of continual learning research, which can bridge this gap and reduce the overall cost of learning by reusing previously trained models, continues to garner significant interest. In this post, we will introduce one of the continual learning research directions presented at CVPR 2022.

In the field of continual learning, methodological research on unsupervised (or self-supervised) situations is commencing in earnest, influenced by the recently active research on self-supervised learning. At this conference, two representative papers ^[5,6] were presented.

(Figure 5) CaSSLe Example

(Figure 6) Comparative analysis between CaSSle (Ours) and supervised fine-tuning

 

The two papers demonstrated experimentally that self-supervised contrastive learning-based continual learning is less prone to catastrophic forgetting than conventional supervised learning-based continual learning. However, [5] differs from [6] in that it considers both self-supervised and supervised situations, whereas [6] is more concerned with self-supervised learning. In addition, [6] proposed a simple idea (CaSSLe) that can aid in learning a better representation in the continual learning process while overcoming catastrophic forgetting using self-supervised learning, as depicted in Figure 5. Moreover, as shown in Figure 6, it has been demonstrated experimentally that this method yields superior results to supervised fine-tuning in continual learning situations.

In addition, CVPR 2022 introduced continual learning algorithms for various domains, such as Visual Search and Semantic Segmentation, as well as Continual Learning for Image Classification, which has been the subject of the most research to date. This makes us believe that continual learning may be getting closer to solving more realistic problems. More information on the continual learning presented at CVPR 2022 will be provided in a separate post (link).

Conclusion

In this post, we’ve introduced CVPR 2022 from the viewpoint of three researchers at LG AI Research. LG AI Research not only demonstrated its technological prowess in the field of computer vision, which is referred to as the “eye of AI,” to the world via a corporate booth but also presented a total of seven papers, including independent research papers, to present outstanding research results. The independent research paper on the outcomes of L-Verse (L-Verse: Bidirectional Generation Between Image and Text), a key technology for implementing large-scale multimodal AI that processes both language and visual information, was selected for presentation in the Oral Session. LG AI Research will continue to push itself to become a leading research institute in global AI technology by actively collaborating with academia in the future.

 

▶LG AI Research Presents Research Achievements at CVPR 2022 and Hosts 'LG AI DAY' (Link)
▶[LG AI Research] LG AI Research Presents Research Achievements at CVPR 2022! CVPR 2022 field sketch (Link)
▶[CVPR 2022]L-Verse: Bidirectional Generation Between Image and Text (Link)