[CVPR 2024] Introduction to Papers Selected for the Best Paper Award and Low-Level Vision Research

As one of the most prestigious international conferences in the field of computer vision and pattern recognition, the Computer Vision Foundation (CVPR) has established itself as an annual opportunity for researchers worldwide to present their latest findings in vision and share insights into technological advances. Notably, CVPR 2024 attracted more than 11,500 submissions, with 2,719 papers accepted for publication, a 26% increase over 2023. More than 12,000 attendees gathered on-site to share their knowledge. In this post, I will introduce the "Generative Image Dynamics" study, which was named Best Paper at CVPR 2024, and my own research in the area of low-level vision.

I. Generative Image Dynamics^[1] : Best Paper Award Winner in CVPR 2024

This year’s CVPR attracted a wide variety of submissions, ranging from traditional computer vision fields such as segmentation and classification, to image and video generation, 3D from multi-view and sensors, medical and biological vision, robotics, and low-level vision. Among these, one of the hottest fields of research is image and video synthesis, which has great potential for commercial applications as well as academic research. Various services, such as image generation models like DALL-E and OpenAI’s Sora, have gained significant attention in the computer vision community and are actively being researched. This trend is reflected in the fact that there were over 1,000 research papers submitted in this field alone. It is especially noteworthy that out of the 11,500+ papers submitted to CVPR, the two “Best Paper” winners were in the field of image and video synthesis.

As the title suggests, "Generative Image Dynamics" focuses on predicting the dynamics of images through generative models. This study specifically aimed to generate videos with natural swaying motion from images of trees, flowers, candles, etc.

This study creates a naturally swaying video from a single image. Natural objects such as trees, flowers, and candles can be made to sway naturally by wind or other external forces, and even better, videos can be made that respond to user input. Inspired by previous studies showing that this natural movement can be represented through harmonic oscillations at different frequencies, it was confirmed that using this method could create naturally repeating videos. This is in contrast to previous studies where the image became inconsistent as the generation time increased. To achieve this, the authors analyzed the image in the frequency domain and used a diffusion model to predict the spectral volume^{[2, 3]}, determining the movement of each pixel in the image within each frequency domain. As learning data, the authors collected 3,000 videos containing natural motion and generated ground truth data by predicting how each pixel moves using the optical flow model ^{[4, 5]}.

Image 1. Generative Image Dynamics^[1]

The study even provides an interactive demo for users to experience. Users can click and drag the image to move it around as if they were touching a real object, and then watch it return to its original position.

▶ Try the interactive demo

Image 2. Motion Prediction Module^[1]

The authors used K=16 output frequencies to predict short to high-frequency motions and, through experimental analysis, found that it was sufficient to use ranges down to 3 Hz. The structure of the model is based on the latent diffusion model^[6], which consists of a variational autoencoder (VAE) model that maps the image into latent space and a U-Net-based diffusion model. The denoising process is applied to the spectral volume rather than the RGB image. The predicted motion in the frequency domain is restored to the target image through the inverse Fourier transform process and a synthesis network. To eliminate holes that could occur in the forward warping process, a pyramid-shaped hierarchical structure proposed in the literature was used^[7].This structure and learning allow us to generate naturally repeating videos and further extend existing research methods^[8], where the spectrum coefficients were represented as a mass-spring-damper system, allowing the generation of videos that looked like an arbitrary force was being applied to an object.

This is just a brief review of “Generative Image Dynamics,” which won Best Paper at CVPR 2024. The study showed that by predicting the movement of image pixels in the frequency domain with a diffusion model, it is possible to generate naturally repeating images. The resulting model exhibited more natural motion than other methods, as intended by the authors, and also responded to user input.

II. Learning to Control Camera Exposure via Reinforcement Learning^[9]

LG AI Research also attended CVPR 2024 and presented a total of five papers. Among them, this post will introduce “Learning to Control Camera Exposure via Reinforcement Learning,” a study in the field of low-level vision. This study aims to obtain better quality images during the initial process of image acquisition before computer vision algorithms come into play. The low-level vision field received close to 600 submissions for this CVPR, but the majority of the work dealt with ISPs such as super-resolution, deblurring, and denoising, with very few papers on camera exposure control.

Recent computer vision studies have shown remarkable results in complex tasks such as recognizing multiple objects from images, estimating depth, and 3D reconstructing the surrounding environment. In particular, when interacting with the real world, such as in robotics, image information plays a crucial role in acquiring a variety of information about the surrounding environment and deciding how to act on a given task based on this information.

But what happens if the quality of the various images acquired in real situations is poor? Obtaining blurry results when taking photos in the dark, backlighting from sunlight, and blurry fast-moving objects are all phenomena we encounter very often in real life. A lot of research has been done to make these low-quality images robust enough to perform a given task, but they still fall short of the results we get from high-quality images.

Image 3. Camera Pipeline Overview^[9]

This post will briefly discuss the motivation for this study by explaining how light reflected from an object actually reaches the image processing algorithm. Light from a light source, such as the sun or an illumination, strikes an object, which absorbs some of it and reflects the rest. The reflected light first reaches the camera’s lens, where it is refracted, passed through an aperture, and converted into an electronic signal by a sensor like a CCD or CMOS. At this point, the shutter controls the amount of light by only allowing it to reach the sensor for a certain amount of time. When light reaches the sensor and is converted into an electronic signal, the sensitivity, known as the gain or ISO sensitivity, is determined.

After light reaches the sensor and is converted into electronic signals, it goes through various processes handled by the image signal processing modules built into the camera to transform it into the images we know. This process involves various image processing modules, such as white balance, denoising, and sharpening, which improve the quality of the acquired image. This entire process can be divided into two parts: exposure control, which includes aperture, shutter speed, and ISO sensitivity, and image signal processing, which is responsible for processing the acquired image afterward. As mentioned earlier, there are limitations to restoring images that have already been acquired, so it is very important to acquire good quality images starting with exposure control.

To this end, commercial cameras have built-in auto exposure modules along with ISP modules, which are designed to respond to a variety of changes to achieve good quality images. However, because these built-in AE modules must run on the camera’s internal processor, they tend to have simple algorithms that make it difficult to respond quickly to external changes.

While this is not a big deal in the context of everyday photography, in fields like SLAM and visual odometry, where continuity of image quality is critical, even a short period of image quality degradation can cause the entire algorithm to collapse. Therefore, ensuring image quality robustness is crucial for applications in fields such as robotics and autonomous driving.

This is where the motivation for this study comes from. Existing research on exposure control has suffered from the complexity of the methods they propose, requiring GPUs^{[10, 11]} or many steps to converge, making them difficult to run in real-time^{[12, 13]}. This study proposed a learning method that utilizes very simple neural networks and reinforcement learning to quickly respond to changes in the input signal to ensure quality. This paper leaves the aperture, which requires external mechanics, at a constant value and allows the trained agent to adjust the shutter speed and gain (ISO sensitivity).

Reinforcement learning is a learning method that can be considered alongside unsupervised and self-supervised learning in situations where it is difficult to obtain the answer label. Finding and labeling the correct shutter speed and gain from a large number of data images is a challenging task. In fact, previous research has performed supervised learning by acquiring a dataset of real-world cars in motion and using the shutter speed and gain from the camera’s built-in AE as labels when acquiring images^[13]. However, this approach has the disadvantage of limited data acquisition and limited light variations. Introducing reinforcement learning eliminates the problem of labeling the data, but it becomes necessary to find the right reward function to guide the learning, as well as to experience different samples during the learning process.

In a typical reinforcement learning environment, these drawbacks can be overcome by learning through a simulated environment. We also considered using simulation but found that the simulator’s images differed significantly from the real world due to the difficulty of getting the right image for different shutter speeds, gain values, and extreme dark or light situations. To address this, we constructed an environment with controllable illumination within a physical darkroom, allowing the agent to experience a range of illumination levels as part of the learning process. We introduced the curriculum learning^[14] method to allow the agent to learn to adjust to relatively easy medium illumination at first, gradually moving to very dark or bright situations, and finally to an environment with dynamically changing illumination values as learning progresses. We also applied augmentation techniques such as rotating, flipping, and cropping some areas of the image to overcome the constraints of the simplicity of the objects in the environment.

Image 4. Training Framework Overview^[9]

The network structure uses a simple 3 FC layer to output the changes in shutter speed and gain. As input to the network, we used the history of intensity along the axis of the input image instead of a complex structure for faster processing. For the reinforcement learning algorithm, we used the sample-efficient Soft Actor-Critic (SAC) algorithm^[15], and the reward function was calculated based on the overall brightness, noise level, and change from the previous frame. This guided the learning process to maintain brightness, reduce noise, and avoid dramatic changes from the previous image.

Figure 5. Experimental Results^[9]
(Related Video : https://www.youtube.com/watch?v=66gjCGSh-mE, https://www.youtube.com/watch?v=YhfJDTtyYqk)

It was found that the trained agent converged about five times faster than the built-in AE module in extreme change situations. When mounted on a vehicle and run in real-time on a laptop in a real-world environment, it was able to acquire images superior to those produced by the built-in AE algorithm without any additional learning. In particular, when we compared the number of SIFT features for use in computer vision applications, we found that, on average, we could get about 40% more features. Even when performing object detection with the pretrained YOLOv5 model^[16], we could detect objects 4-5 frames faster than with the built-in AE.

This post reviewed studies in image and video synthesis and low-level vision that were selected as Best Papers at CVPR 2024. LG AI Research continues to conduct various studies to overcome the limitations of visual data and apply AI to various applications, including studies utilizing generative models. AML (Advanced Machine Learning) Lab is working on anomaly detection using diffusion models, while Multimodal Lab is focusing on generating images from text. Furthermore, Data Intelligence (DI) Lab is using generative models for combinatorial optimization and time series prediction, and Materials Intelligence (MI) Lab is applying them to synthesize new proteins or predict the properties of materials. LG AI Research plans to continue advancing research in various fields and creating achievements, thereby transforming our lives and innovating industries through AI technology.