[CVPR 2023] Looking at CV Research Trends and the Future of Image Captioning at CVPR 2023

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023, the world’s largest computer vision conference, was held for 5 days from June 18th to 22nd in Vancouver, Canada. At CVPR 2023, LG AI Research had a chance to review research trends in the computer vision (CV) field through various papers and held a workshop titled “NICE: New frontiers for zero-shot Image Captioning Evaluation” to interact with world-class AI experts. Let's hear from Soonyoung Lee and Pyunghwan Ahn of the Vision Lab about noteworthy papers from CVPR 2023 and a review of the NICE workshop.

 

Photo 1. The Vancouver Convention Center where CVPR 2023 was held

 

[Vision Lab Soonyoung Lee] CV research trends examined through CVPR 2023 presentations

A total of 9,155 papers were submitted to CVPR 2023. The fact that the number has increased by 12% from the previous year shows that both industry and academic interest in computer vision and AI is still at an all-time high. 2,359 papers—or one-quarter of the total—were accepted, and they excellently reflect the current research trends and directions in computer vision study. Here are some titles of award-winning papers that were released this year and reflect current trends.

 

Best Paper: Visual Programming: Compositional visual reasoning without training[1] Best Paper: Planning-oriented Autonomous Driving[2] Honorable Mention: DynIBaR: Neural Dynamic Image-Based Rendering[3] Best Student Paper: 3D Registration with Maximal Cliques[4] Honorable Mention (Student): DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation[5]

 

Various topics such as autonomous driving, NERF (Neural Radiance Field), classic 3D registration problems, and multimodal can be seen. Through this, CVPR 2023 shows an unbiased - not focused on a certain topic or issue - research trends, which seems a desirable phenomenon. But at the same time, it infers that there is no new technological innovation which could become a mainstream.

Amidst these diverse themes, I was particularly interested in the paper “Visual Programming: Compositional visual reasoning without training^[1]”, which won the Best Paper Award and solved computer vision issues through a new attempt at the Large Language Model (LLM).

 

LLM, solving vision issues with human logic

“Visual Programming: Compositional visual reasoning without training^[1]”proposed the idea of defining various modules to solve the Vision task, and determining, with LLM, the order in which these modules should be performed and which parameters should be put in place.

Up until now, the research direction of existing AI technology has been directed toward securing high performance for individual tasks (Detection, Segmentation, VQA, Captioning, etc.). However, actual issues are very rarely composed of one isolated task and are often a mixture of various tasks, or tasks that are difficult to define. Therefore, the most basic approach to solving the issues in the field is to build a specialized model by building data specific to said issue. Obviously, this process takes time in building training data and requires a lot of resources to train the model, and such a model learned with such effort tends to become an inefficient structure that cannot be applied to other issues.

This paper demonstrates that task-specific issues can be solved by creating optimal combinations of previously-developed models and techniques, rather than creating a new model based on these points. The core idea of this paper is to make previously developed models and technologies into modularized function blocks and use GPT-3 as a scheduler to call these function blocks.

 

Figure 1. Structure of VisProg ^[1]

 

Figure 1 shows the structure of the technology presented in this paper. Interactive indicators, the strength of GPT, are available, so once you order the desired task in a sentence, a series of procedures and parameters are created to solve it. This is much like a human writing code that solves a given issue. That’s why the author named this technology Visual Programming (VisProg). Through this, various vision issues such as those below can be solved with existing models and computer vision technologies, without new models and training data.

 

Figure 2. Various examples of VisPro^[1]

 

Figure 2 shows various examples of VisProg. The fact that LLM is not just a language model, but a model that simulates human logic, leaves quite an impression. The diligence of implementing multiple models at the same time and engineering them to seamlessly connect was also quite impressive.

Technology can be more meaningful when used in real life. There are already many task-specific models out there that work quite well. To make them even more practical, these models will need to be connected and tuned to solve complex, vaguely defined issues that are a necessity in real life. One example is the problem-solving abilities of GPT-4 in combination with plugins. One can make a careful guess that this direction in research will become one axis in future computer vision research.

At the LG AI Research Vision Lab, various attempts are being made using multimodal, a convergence of vision, and language problems. Representative tasks include the Image-to-Text and Text-to-Image tasks shown with the EXAONE Atelier, as well as tasks that extract data from diagram elements (charts, tables, etc.) in documents. Through multimodal technology that uses both visual and language information, astounding progress has been made in solving issues that cannot be solved with existing computer vision technology alone. LG AI Research Vision Lab will continue to lead in new research trends that can solve multimodal tasks scattered inside and outside LG affiliates.

 

[Vision Lab Pyunghwan Ahn] Review of the NICE workshop

In addition to annual paper presentations, CVPR has been discussing various issues related to specific research topics, research environments, and social impact through workshops and tutorial sessions. At this CVPR 2023, LG AI Research held the NICE workshop together with Shutterstock and Seoul National University. In addition to the announcement of the results of the zero-shot image captioning challenge that had been held prior to the conference, various topics related to image captioning were discussed as well.

 

The NICE workshop, a forum for discussing overcoming the limitations of image captioning

 

Photo 2. Photo from the NICE workshop

 

Image captioning is a task that analyzes a given image and describes the elements of the image in natural language, and is an important foundation for image understanding in the field of computer vision. Despite the recent rapid development and growth of multimodal models, there are still many limitations that are difficult to overcome in the field of image captioning. Due to the limitations of simple text comparison metrics used as evaluation criteria for image captioning results, the quantitative evaluation of AI models remains challenging.

For example, even if hallucination occurs so that the contents not included in the image are presented in the predicted caption, there are cases in which it is still evaluated as a good caption based on various metrics^[6]. Furthermore, depending on the distribution of data, bias issues that associate certain groups of people with certain behaviors or characteristics, regardless of the input image, are often found. Such issues require in-depth discussions by experts in each field, as it is difficult to solve them only with technology in a specific field. Through NICE workshop, LG AI Research provided an opportunity to share research results and directions related to these issues.

 

Opening Talk & Challenge result announcement session

 

Photo 3. CVPR 2023 NICE Challenge 1st to 4th place winners

 

The NICE workshop opened with an opening talk by Professor Kyoung Mu Lee of the Artificial Intelligence Institute of Seoul National University. After this, the NICE challenge was introduced and results were shared.

This challenge, which ran from January to April of this year, evaluated the image captioning performance of a dataset constructed from approximately 30,000 images and captions for each image. Image and caption data were provided by Shutterstock, the world’s largest image stock platform. In this workshop, the award ceremony was held with the 1st to 4th place research teams, who showed excellent image captioning performance, being invited. The ceremony was attended by Professor Kyoung Mu Lee and Professor Bohyung Han of Seoul National University, and Alessandra Sala, Senior Director of AI and Data Science at Shutterstock. In addition to the award ceremony, the team from Nanjing University of Science and Technology, who took the 1st place, and the team from KAIST, who took the 2nd, also presented their award-winning algorithms.

 

Hearing insights from world-renowned scholars, the Invited Talk Session

Next, in the Invited Talk session, researchers who have demonstrated good results in the field of image captioning around the world were invited to share their insights on the limitations of current multimodal models and future research. In the first part of the session, Anna Rohrbach (UC Berkeley) presented research results on high-accuracy image captioning through the extraction of text information that is highly relevant to a given image and on accurate evaluation method. Cordelia Schmid (INRIA, Google Research) shared the latest research on video captioning. Continuing, Hamid Palangi (Microsoft Research) presented the limitations of the image captioning model with a simple structure and methods for solving these limitations, such as object-to-object relationship learning, multitask learning, and data collection for specific purposes.

 

Photo 4. CVPR 2023 NICE Workshop, Jack Hessel’s presentation

 

In Part 2, Yusuke Hirota (Osaka University, supervised by professor Noa Garcia)’s presentation dealt with the societal bias problem in AI models and how to solve it, while Jack Hessel (Allen Institute for AI) shared interesting research on topics including learning relationships in time order in video captioning, reasoning about context beyond image description, and accurate image captioning in unusual situations. To wrap the workshop up, Honglak Lee, Chief Scientist of AI at LG AI Research, talked about the limitations of zero-shot image captioning models presented so far and the direction and goals of future image captioning research.

 

 Photo 5. CVPR 2023 NICE Workshop group photo

 

Based on the research results discussed at CVPR 2023 and the achievements gained from the challenge and workshop, LG AI Research is conducting further research in advanced image captioning. As multimodal models recently show the ability to perform various tasks beyond simple image captioning based on the combination of vision and language models, LG AI Research will also continue to seek for the expansion of tasks and they may be included in the next workshop and challenge.

 

▶CVPR 2023 - Watch the NICE Workshop sketch video (Link)