Hyunmin Lee
2022.10.18
Minsoo Lee
2022.10.18
[GTC 2022] Part 2: The Latest AI Research Trends and Insights
|
NVIDIA GTC is a global AI conference focusing on artificial intelligence, computer graphics, and machine learning. NVIDIA GTC 2022 was held online from September 19 to 22 where researchers and business leaders advancing the latest technologies actively exchanged and communicated on topics such as large language models, digital twins, and human AI. Hyunmin Lee and Minsoo Lee from Vision Lab, LG AI Research, who took part in the online conference, talk about GTC 2022. |
Personalized Image/Language Model
Figure 1. The output of DALL-E Creating New Images Using Sentences as Input[1]
OpenAI unveiled DALL-E[1] in January 2021, a model that generates images based on text input. DALL-E has attracted a great deal of attention by not only producing realistic images that express a given sentence about any subject, but also by producing imaginative images that do not exist in reality. This technology, which generates images based on the sentences entered by the user, is anticipated to have a significant impact on industries such as art, fashion, and advertising.
Figure 2. Example of Personalized Image/Language Model Proposed by NVIDIA[3]
NVIDIA argues that personalized models are required for these image/language models to provide enhanced user convenience. For instance, if you input the sentence “A Cat themed lunchbox” into the existing text-to-image generation model, you can expect an image of a lunchbox featuring a cat. However, many users want to generate a cat image of their own or their favorite cat characters, as opposed to random cats. NVIDIA proposes a personalized image/language model to meet these users’ needs. Figure 2 shows an image of a lunchbox with a cat created by providing an additional image of a cat in addition to the model.
Figure 3. How the Images-to-Word Mapper Works[3]
To implement such a personalized model, this study proposes an images-to-word mapper and uses it together with the existing image/language model. This network maps the user-supplied concept image to the text embedding space. The concept image embedding mapped to the text embedding space replaces the token embedding of any word in the text (Cat replaced in the example of Figure 2 and My teapot in the example of Figure 3) so that the feature produced by the text encoder can express the user-supplied concept image. The user's desired image is then generated using the text-image generation model with the input of personalized text embedding.
Figure 4. Example of Personalized Model Applied to Text-Image Search[3]
A personalized model can also be applied to text-image search in addition to text-image generation. Text-Image search identifies the image in the database that most closely resembles the text entered by the user. Text embeddings are extracted from the user's input sentence, and the closest image is located by calculating the distance between the image embeddings extracted from each image in the database and the extracted text embeddings. In Figure 4, the phrase “Standing with My Skirt on a Stone Pathway” is used as input for an image search. In this case, the user's convenience can be enhanced by providing the user's concept image of the model in place of the phrase “My Skirt.” For this purpose, as in the case of text-image generation, the concept image is mapped to the text embedding space using the images-to-word mapper. A personalized model can be applied to the text-image search task by replacing the token embedding corresponding to “My Skirt” with this embedding.
Instant NeRF[4]
Figure 5. 3D Representation of NeRF[5]
Neural Radiance Fields (NeRF)[5] have become the hottest topic in computer vision since its announcement at ECCV in 2020. NeRF is a method that successfully completed the Novel-View Synthesis task, which restores 3D information about a scene using images captured from various views of the particular scene and renders the scene from a previously unseen view. NeRF uses the neural implicit representation to represent 3D space. To be more precise, it takes as input the expression (x,y,z,,) for 3D coordinates and observation direction, and uses MLP to express RGB values and volume density at the relevant location and direction. NeRF renders the scene realistically with a high compression ratio in terms of storage space due to this neural implicit representation. However, optimization time and runtime performance continue to be significant NeRF limitations. Given that more than 80 million MLP operations are required to render a single 800×800 image, it takes more than 10 hours to optimize NeRF and about 30 seconds to render a single image, making real-time rendering impossible. Methods such as combining NeRF and octree structures[6] or directly storing 3D information in voxel structures without using MLP[7] have been proposed to circumvent these limitations.
Figure 6. Instant NeRF Learning in Seconds[4]
Earlier this year, NVIDIA announced Instant NeRF[4], which has attracted attention due to its remarkable performance improvement. The optimization time of NeRF, which had previously taken more than several hours, was reduced to less than a few seconds, and it was able to render more than 60 images per second. This research was selected as the Best Paper of SIGGRAPH 2022.
Figure 7. Instant NeRF Schematic[4]
The main concept of Instant NeRF is Multiresolution Hash Encoding. (1) of Figure 7 shows the encoding of x at two resolutions when x is input as a query point. To encode continuous point x, it uses an interpolation of discrete points near x at various resolutions. Looking at the encoding at the resolution shown in light blue, the value of x can be determined by interpolating the four corner points in light colors (0, 1, 4, and 7). Similarly, the value of x can be interpolated from four corner points when encoding at the resolution indicated in red. It was said that performance could be significantly enhanced if the query point values were encoded by interpolation of discrete points (corner points) at different resolutions. Following this, the values of each corner point at various resolutions are stored in the hash table, and these values are updated as learning progresses. Finally, after concatenating the encoding information of the query point from multiple resolutions, it is then used as an input to a thin MLP to obtain RGB values and density. In the case of the previous NeRF, calculating the values corresponding to the query point required a significant amount of computation time because it traversed a dense MLP; however, Instant NeRF retrieves the value stored in the hash table and obtains the final value through a simple interpolation operation and thin MLP, thereby drastically reducing the computation time.
Neural VDB[8]
Figure 8. NeuralVDB[8] Compressing Existing VDB Storage Space by More than 50 Times
3D modeling data is utilized frequently in numerous industries, such as the gaming, advertising, and cultural sectors. However, 3D modeling data requires a substantial amount of storage space in comparison to other types of data. In particular, it is essential to adopt an efficient data structure for storing 3D volumetric data due to its sparse nature. Consequently, several compact data structures have been proposed over the years in order to effectively represent sparse volumetric data. Among these data structures, VDB[9] has recently gained widespread acceptance. In the present research, by combining VDB with Neural Network, the NeuralVDB[8], which can compress the storage space of existing VDB by 10 to 100 times or more with minimal visual artifacts, was proposed.
Figure 9. VDB Schematic[8,9]
VDB is a hierarchical tree structure that represents high-resolution sparse volumetric data. All levels of the VDB tree structure contain mask value information indicating whether a node is active or inactive, allowing sparse data to be expressed effectively. The voxel value is stored at the leaf level, while the upper node stores the tile value that serves as a child pointer. In general, a four-level tree is utilized, and the lower-level nodes are sequentially connected to the root node, with their respective sizes being 323, 163, and 83. Internal nodes (levels 1 and 2) contain active mask and child mask information, each of which indicates whether a tile's value is active or connected to a child node. The active mask information of a leaf node indicates whether or not the voxel value is active.
Figure 10. NeuralVDB Schematic[10]
With the rise of deep learning, numerous studies have been conducted to represent 3D volumes using neural networks such as SDF[11] and NeRF[5]. Since the number of parameters of the neural network is typically much smaller than the number of nodes representing the volume, a high level of compression performance can be expected. In addition, with the development of feature mapping/encoding techniques such as SIREN[12] and Fourier Feature Mapping[13], it is now feasible to fit a neural network with minimal information loss. Similarly, in the current study, a new sparse volumetric data representation called NeuralVDB that combines the strengths of a neural network and VDB was proposed. NeuralVDB integrates mask and value information from one or two levels of the existing VDB structure into the neural network. With x as input, the network is trained to predict mask information or value information at each level. In this combination, VDB provides efficient hierarchical segmentation of 3D data, whereas the neural network enables 3D data compression with astounding performance. Moreover, it enables the VDB tree to concentrate on topological information at the parent node level, while the neural network encodes voxel features and minute topology and value information at the child node level.
Figure 11. Comparison Between VDB and NeuralVDB in Terms of the Number of Nodes, Parameters, and Storage Space[10]
In the current research, a structure that replaces one lower level of VDB with a neural network (NeuralVDB) and a structure that replaces two lower levels of VDB (NeuralVDB with Hierarchical NN) were proposed. Figure 11 is a table displaying the storage space of VDB and two structures of NeuralVDB. NeuralVDB was able to reduce the size of the original VDB to 6.268%, while NeuralVDB with Hierarchical NN, which encodes two lower levels, reduced the size to 1.484%. However, the greater the compression level, the longer it takes to access data compared to conventional structures.
3D Human Pose Estimation
In this session, we explain 3D human pose estimation, which is the study of determining the 3D joint position of a human body in an image. This research field is challenging due to the ill-posed target problem and the difficulty of collecting learning data. Specifically, it is an ill-posed problem since it extracts 3D joint information from a 2D image. In addition, it takes a long time to collect 3D joint positions because the data is collected in a studio with MoCap equipment and there is a domain difference between “in-the-wild” images.
Figure 12. Prediction of Human 3D Joint Positions in In-the-Wild Images[15]
In addition to using the 2.5D pose expression method as an intermediate step, NVIDIA proposes a weakly-supervised method[14] that uses unlabeled multi-view images instead of 3D joint positions that are difficult to create as learning data in order to overcome this challenge. Multi-view data[16] has the benefit of being simple to capture in an in-the-wild environment as it does not require a separate hardware device to create a dataset, even though the 3D joint position is not labeled.
Figure 13. A Model that Learns 3D Poses Using Only Unlabeled Multi-View Images and 2D Pose Data
When the 3D pose of a joint (j) is defined as , the 2.5D pose is expressed as . is the coordinates on the 2D image and, is the distance relative to the root, which is . The 2.5D expression method is distinguished by its relative distance from the 2D pose, which confers several advantages. First, 2D pose data that can be annotated easily from in-the-wild images thus can be used for supervision, and, as demonstrated by Iqbal et al.[17], 3D poses can be reconstructed in a fully differentiable way by using 2.5D pose.
The overall learning process is depicted in Figure 13. Given a single image as input, the network outputs a 2.5D pose. We know that the 3D joint position of a person in a multi-view image should be the same regardless of the camera's point of view and this information is used to provide 3D supervision. When the predicted 2.5D pose from the multi-view image is reconstructed in 3D and aligned via rigid transformation, we can give multi-view consistency loss that the 3D joint position must be identical. In addition, a 2D loss is provided using a 2D pose dataset, and a bone-length loss that brings the predicted bone length closer to the average value permits faster convergence.
Figure 14. Experimental Results Comparing with the SOTA Method in the Human3.6M[18] Dataset
Figure 14 depicts the experimental results in the Human3.6M dataset[18]. The table is separated into semi-supervised and weakly-supervised categories based on whether the 3D pose data labeled in the Human3.6M is used as supervision. In the case of semi-supervised classification, the 3D data of one object (S1) out of five objects in the dataset is utilized, while the other four objects (S5, S6, S7, and S9) are used as weak supervision. In contrast, in the case of the weakly-supervised method, the 3D pose is learned without using the 3D pose data annotated in Human3.6M by employing the multi-view consistency loss mentioned above. Ours-H36M+MPII-3D+Mannequin showed the best performance in the semi/weak supervision categories, demonstrating that 3D poses can be successfully learned from multi-view images without 3D data, which is difficult to acquire in an in-the-wild environment.
Conclusion
We have examined the research presented during the GTC 2022 session. Looking at the research introduced by NVIDIA at the conference, it can be seen that the company selected and presented research from three major categories of the current AI research trends: Large Scale Vision/Language Model, Neural Rendering, and Digital Human. In addition to learning about the most significant research findings, we were able to gain insight into whether or not service users could be offered greater practicability and convenience. As AI is gradually applied to a growing number of industries and begins to provide direct benefits to people, LG AI Research will continue to carry out relevant research and share our insights.
- 참고
[1] https://openai.com/blog/dall-e/
[2] Ramesh et al., Zero-Shot Text-to-Image Generation, arXiv, 2021. [3] NVIDIA GTC 2022, Insights from NVIDIA Research, https://register.nvidia.com/flow/nvidia/gtcfall2022/ attendeeportal/page/sessioncatalog/session/1656017771434001ZAZg [4] Muller et al., Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, SIGGRAPH, 2022. [5] Mildenhall et al., NeRF: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020. [6] Yu et al., PlenOctrees for Real-time Rendering of Neural Radiance Fields, ICCV, 2021. [7] Yu et al., Plenoxels: Radiance Fields without Neural Networks, CVPR, 2022. [8] Kim et al., NeuralVDB: High-resolution Sparse Volume Representation using Hierarchical Networks, arXiv, 2022. [9] Ken Museth, VDB: High-resolution sparse volumes with dynamic topology, TOG, 2013. [10] NVIDIA GTC 2022, High-resolution Sparse Volume Representation Using Hierarchical Neural Network, https://register.nvidia.com/flow/nvidia/gtcfall2022/attendeeportal/page/sessioncatalog/session/1658516935505001BqIc [11] Park et al., DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation, CVPR, 2019. [12] Sitzmann et al., Implicit Neural Representations with Periodic Activation Functions, NeurIPS, 2020. [13] Tancik et al., Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, NeurIPS, 2020. [14] Iqbal et al., Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild, CVPR, 2020. [15] https://www.youtube.com/watch?v=VzYkvQ9FgBg [16] Li et al., Learning the depths of moving people by watching frozen people. CVPR, 2019. [17] Umar Iqbal et al., Hand pose estimation via 2.5D latent heatmap regression, ECCV, 2018. [18] Ionescu et al., Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments, TPAMI, 2014.