[CVPR 2022] NeRF - Representing Scenes as Neural Radiance Fields for View Synthesis

The diverse application of NeRF, which has had a significant impact since the 2020 ECCV, was one of the highlights of this year's CVPR. NeRF^[1] is a new method for completing the View Synthesis task, which uses images captured from multiple angles of an object to generate its appearance when viewed from a previously unseen viewpoint. More than 50 papers at CVPR 2022 presented NeRF-based methodologies. In addition, NeRF was mentioned in the titles of a total of 29 papers, of which eight were chosen for oral presentations (top four percent of submissions). This blog post will delve deeper into what NeRF is and how it was primarily utilized at CVPR 2022.

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

View Synthesis is one of the major challenges in computer vision where we must accurately comprehend the 3D structure of an object solely based on 2D images. NeRF obtains the volume(density) and color from a certain ray into the scene, which are afterwards integrated to compose an image. Early methods for 3D rendering where color values are directly stored in 3D units (e.g., voxels) suffer from degrade rendering quality and large memory. NeRF is a revolutionary model that can effectively resolve these issues.

The “ray” from a specific perspective can be expressed using the D expresion (x,y,z,θ,$\varphi$), where x,y,z are the three-dimensional coordinates of the point and θ,$\varphi$ are the direction of the light source.

(Figure 1) NeRF Concept Diagram^[1]

(Figure 2) NeRF Learning Schematic^[1]

NeRF is a simple MLP structure that receives cross-sectional images of an object captured from multiple viewpoints, specifically the 5D expression (x,y,z,θ,$\varphi$) for the above viewpoint, and generates the RGB values and volume density σ of the position within the view. Here, volume density can be interpreted as being density or opacity. This simple structure of NeRF has demonstrated superior performance compared to the previous state-of-the-art view synthesis method. View synthesis is now possible with only the captured image and camera data at the time of shooting (including both extrinsic and intrinsic parameters such as camera pose).

NeRF in CVPR 2022

The NeRF methodologies of the diverse topics presented at this year’s CVPR are categorized into four groups. We will describe how each NeRF is being utilized.

1. NeRF from single images

Previously, NeRF synthesized new views by combining multiple images and camera poses. At CVPR 2022, a method for synthesizing new views from a single view per object was presented. It became a hot topic as it demonstrated excellent performance despite the difficulty of obtaining multi-view data for the same object. This method, which was also introduced at LoLNeRF^[2] and Pix2NeRF^[3], is intended to express a view from an arbitrary angle with a single image after training NeRF with the view fixed in a common pose. NeRF can be learned with only a shared generative model and an approximate camera pose, even if there is only a single view of various classes of objects. Moreover, with the remarkable development of NeRF, it is now possible to defocus objects by generating depth from a single natural image^[4].

(Figure 3) AR-NeRF depth estimation and defocusing function^[4]

2. Beyond the Limits of NeRF

Under this category, we will delve into works in CVPR that aimed to overcome the various limitations of NeRF including slow optimization time and vulnerability to noise. By combining Point Cloud with NeRF while maintaining its high quality, diverse techniques, such as overcoming the slow rendering speed per scene^[5], utilizing depth supervision already known from camera poses^[6], and combining the traditional 3D reconstruction methodology based on TSDF^[7], were presented.

Methods using regularization to overcome the limitations of NeRF also stood out. While the existing NeRF accurately renders objects, it yields blurry results for backgrounds with unclear boundaries and has very severe visual artifacts without a large quantity of multi-view images and camera pose training. Using regularization terms, techniques such as Mip-NeRF 360^[8] and RegNeRF^[9] resolved this phenomenon, and in the case of GeoNeRF^[10], NeRF learning can be made more efficient by solving this problem with consistent geometry. As in the previous IBRNet^[11], a method has been proposed for high-quality human pose view synthesis with sparse multi-view input using pixel-alignment features and pose-embedded non-rigid deformation^[12].

Scalability was also impressive. In the case of Block-NeRF^[13], the demonstration video drew the interest of the participants^[14]. In Block-NeRF, it has been demonstrated that when reconstructing a city-level scene using NeRF, applying the learned NeRF to each block individually by decomposing the entire scene into blocks is highly effective.

(Figure 4) A screenshot from a Block-NeRF demo video^[14]. By adjusting the shooting time, such as day or night, a variety of perspectives of the city can be rendered.

In addition, studies such as NeRFReN^[15] proposed methods for rendering complex reflections in glass or mirrors, which cannot be rendered accurately using conventional NeRF.

(Figure 5) NeRFReN is capable of rendering extremely complex scenes, such as objects reflected in a mirror^[15].

3. Editing with NeRF

Recently, controllable manipulation techniques based on a latent representation within a generative model have been in the spotlight. The papers that combine this manipulation with NeRF constitute the third NeRF topic. From CLIP-NeRF^[16], which deals with manipulation of NeRF using CLIP and multimodal text-and-image, to NeRF-Editing^[17], which presents methodologies that allow users to perform desired deformations in a scene; specific attributes, such as closing the eyes or opening the mouth; and CoNeRF^[18], which demonstrates astounding results where the toy car transforms or even controls the sync of the metronome, were presented.

(Figure 6) With CoNeRF, it is possible to make the eyes and/or mouth open and close,
as well as create a transformation process for a transformable toy^[18].

In addition, a method for creating a new viewpoint with a new style (Stylized-NeRF)^[19] was presented.

(Figure 7) Using Stylized-NeRF, it is possible to generate novel views in the desired style^[19].

Contrast is an essential element that cannot be omitted from photography. Original studies, including HDR-NeRF^[20] and NeRF in the Dark^[21], which use NeRF to create scenes with varying exposures — high dynamic range and low dynamic range — were presented as well.

(Figure 8) HDR-NeRF-generated images with varying levels of contrast^[20].

(Figure 9) Photographs that focused on different objects in a dark environment were rendered using NeRF in the Dark^[21].

4. Human with NeRF

Applying NeRF to humans is the essence of editability. The methods using NeRF (FENeRF^[22] and RigNeRF^[23]) to control the controllability of facial expression, the various attributes of a person (e.g., hair color, teeth spacing, wrinkles), and the pose of the head, which has been attempted in the past numerous times, were remarkable. HumanNeRF^[25], which renders a full viewpoint human body motion video from a single viewpoint YouTube video, and HeadNeRF^[24], which uses NeRF instead of the existing 3D textured mesh to control the attributes of a human head, were impressive studies presented at CVPR 2022. (This is an entirely different methodology, despite sharing the same name as the HumanNeRF sparse input method described previously.) HumanNeRF has been implemented to accurately reproduce not only the pose's viewpoint, but also minute details such as clothing folds and facial features.

(Figure 10) Through FENeRF, it is possible to create faces with varying viewpoints and individualized features^[22].

(Figure 11) With RigNeRF^[23], it is possible to render faces with free expression and angles.

(Figure 12) HumanNeRF enables the rendering of a person from any perspective from a monocular YouTube video^[25].

We examined various NeRF-based methodologies that were presented at CVPR 2022. It appeared that the primary keyword of the NeRF study of last year was ‘an attempt’. Rather than studying how to model NeRF efficiently using a single model as opposed to a scene-based model, attempts were made to overcome the limitations of NeRF, which yielded many interesting studies. Following last year, we expect many studies on NeRF to be carried out this year as well.

[References] NeRF at CVPR 2022 (Link)

▶[CVPR 2022] LG AI Research joins CVPR 2022, the Mecca of Vision Research (Link)
▶[CVPR 2022] Research Trend of Continual Learning (Link)