Opening Up New Possibilities for AI, Lab Leader Soonyoung Lee of the Multimodal Lab

With advances in artificial intelligence, we're seeing unprecedented innovation in almost every field. In particular, multimodal AI, which learns and thinks with different data such as text, images, video, and audio together, is expected to enrich our daily lives in amazing ways.

LG AI Research’s Multimodal Lab is a team that aims to bring new user experiences to life by unraveling and modeling the information intertwined between visual and language data. We talked to Soonyoung Lee, the leader of the Lab, about current trends in multimodal AI and what’s coming in the future. You can also check out the story of Multimodal Lab on the LG AI Research YouTube channel.

Spotlight on the Importance of AI in Vision

Lab Leader Soonyoung Lee, who studied computer vision, earned his PhD by studying the problem of finding correspondence matching in multi-view images. He joined LG AI Research in February 2021 because he realized that AI technology and AI solutions are essential in the field of vision research, and wanted to learn and research the latest AI technologies. He particularly emphasized being "really satisfied" at LG AI Research. He highlighted the advantage of being able to closely observe and collaborate on AI research conducted in various fields.

Q. What kind of work do you do at LG AI Research?

“After joining LG AI Research, I’ve been working on a project on Deep Document Understanding (DDU), a technology that understands non-text elements such as images, tables, charts, and chemical formulas within documents. In order for machines to utilize these large amounts of data, they require very complex processing. My fascination with extracting and compressing the necessary information from massive amounts of data led me to focus on vision research.”

* DDU (Deep Document Understanding): Vision technology that analyzes data quickly and accurately by analyzing not only the text of specialized literature such as papers and patents but also visual elements such as formulas and tables

Generative Multimodal AI Technology Trends

Q. What are the trends in multimodal technology that you see in the field?

“Multimodal technology is advancing at a very rapid pace. The first trend in multimodal technology is the expansion of image generation technology. While the technology has traditionally been limited to generating still images from text, it is now expected to expand to the time axis and create short-form or long-context videos. They’re also hoping to expand modalities. It's expected to evolve into a multimodal generative model that can generate not only images and video but also 3D and audio.

The second trend is the field of translational medicine. This is a technology that combines not only images but also various clinical trial data and gene sequence data to enable more accurate diagnoses while also allowing for the design of efficient and cost-effective clinical trials.

The third trend is embodied AI. Technology that uses multimodal models, which take in images, speech, and other inputs to collectively understand the environment and control an agent to perform a given task, is expected to gain traction in the future.”

*Long context: A technology that increases the accuracy of answers by providing longer context for LLMs(Large Language Models) to understand

Q. What areas of research are conducted at the Multimodal Lab?

“The Multimodal Lab is currently focused on two areas of research. The first is the field of Generation, specifically image generation AI technology that generates an image from a given text. We’re also focusing on the field of image editing, where a generated image can be changed or restyled according to the instructions of the user.

For example, when creating a design for a cosmetic container or package, you need images for inspiration. So, we’re currently collaborating with LG H&H to help create images through LG AI Research’s image generation technology.

We’re also researching Image understanding. This is also known as Image captioning and is a technology that generates sentences that accurately and richly describe an input image or keyword text that represents its main components. By combining vision features and LLM, we aim to create a model that is free of hallucinations that make untrue information plausible.”

LG AI Research hosts the "NICE (New Frontiers for Zero-shot Image Captioning Evaluation)" challenge and workshop at CVPR 2024, the world's top vision conference, to share image captioning technology with global researchers. This event is co-hosted by LG AI Research, Seoul National University, and AWS.

This workshop features two challenges. The first is an “image captioning challenge,” which involves coming up with a caption that best describes an image. The second is a “captioning re-ranking challenge,” which involves listing the captions in the order that best describes the image presented. This workshop provides an opportunity to share zero-shot image captioning technology and showcase LG AI Research's technical leadership.

▶ NICE Challenge & Workshop : (Link)

▶ AI Image Captioning : (Link)

“The second area of research that we are focusing on is medical data analysis. Traditionally, medical data research has been focused solely on imaging. However, recently, multimodal technologies have been used to develop models that combine various clinical trial data or even gene sequence data. LG AI Research collaborates with various global pharmaceutical and clinical trial companies on tasks such as biomarker discovery and subject classification.”

The Secret to the Multimodal Lab’s Teamwork and Ultimate Goals

Lab Leader Soonyoung Lee aims for a research culture in which individuals can grow together by coaching and spreading their expertise to their colleagues.

“Our researchers at Multimodal Lab have their own expertise. A pair researcher structure, assigning two or more researchers in a single task, leads researchers not only to share their expertise and grow together, but to work more stably even when one researcher is performing other priority tasks or is absent.”

Q. What goals do you want to achieve with the Multimodal Lab in the future?

“Our goal is to improve performance in the field of generation and spread our technology to affiliates and external operators. To do this, I believe that we'll need to study not only the performance of AI models but also other underlying techniques such as prompt engineering and data refinement. In addition, in the medical field, our goal is to create a medical foundation image model that can be applied to a variety of tasks. By applying it to a variety of tasks, our goal is to make LG AI Research a global powerhouse in the medical field.

Ultimately, our goal is for LG AI Research to be globally recognized for multimodal research based on this technical leadership and to become a place where top talent can come and research together.”