The New Hwadamsup: Image Understanding & Generation through LMM

Image 1. The New Hwadamsup Project

Recent advances in AI technology are rapidly changing the form of art and the creative process. By analyzing vast amounts of data and learning patterns, AI creates new forms and styles that artists have never imagined. AI is more than just a tool, it is also a creative partner, suggesting new directions that artists have never imagined and playing an important role in implementing creative ideas. This allows artists to save more time and energy throughout the creative process.

The New Hwadamsup Project (Image 1) is a great example of this successful human-AI collaboration. AI has revolutionized the creative process by working with humans to choose a topic and recommend visual and audio effects to match this topic. LG AI Research’s EXAONE image understanding and generation technology were combined to recommend various elements for Hwadamsup’s media art, and the art was created based on these.

This blog post introduces what LG AI Research is researching on Image Understanding & Generation technology, which directed Hwadamsup’s media art, and how it has been applied in practice.

 

What Is Image Understanding?

Image understanding refers to the task of generating answers to various questions from users based on the understanding of images, and it is usually performed using multimodal models called vision-language models (VLM), large multimodal models (LMM), multimodal large language models (MLLM), etc. Image-based question answering, as demonstrated by GPT-4V^[1] is the most prominent example. The most representative of these LMM models is LLaVA^[2].

Image 2. LLaVA structure^[2]

The structure of LLaVA is shown in Image 2. A given image is changed into a feature containing vision information through a vision encoder and then converted into a feature suitable for LLM through a projection layer. After that, it is aligned with the user’s question information and input to the LLM to obtain the desired answer. This structure has been successfully applied to the LMM field and is used as the basic structure in various LMMs after LLaVA.

The Pros and Cons of Datasets

This structure of LLaVA is possible thanks to the visual instruction tuning data presented in the LLaVA paper. The existing Image-Text Pair dataset simply provides descriptive information about the image. However, LLaVA proposed a dataset that contains information about complex questions and reasoning tasks as well as simple descriptions of images, which significantly improved the performance of the LMM. This approach was also applied in Share-GPT4V^[3], where various instruction tuning datasets were proposed and used to train several LMMs.

However, these instruction tuning datasets have a drawback. Most datasets rely on GPTs to generate data in the hundreds of thousands to millions. While this GPT-generated data is available for research, it is not free from licensing issues. To escape from these constraints, we proposed an LMM with a different structure.

Features of EXAONE LMM

Image 3. Structure of EXAONE LMM

The structure of the proposed EXAONE LMM is shown in Image 3. First, image information is derived from the image through two vision modules: a captioning module, which outputs a description of the image, and detection, which expresses the location information of objects in the image, as shown in [4]. The SLM (Small Language Model) organizes this information into a single sentence to complete the description of the image. This, along with the user’s question, is used as input to the LLM to generate the final answer.

The core of EXAONE LMM is to use the LLM as it is without training. Unlike existing models that use a complex instruction tuning dataset for end-to-end training, it only requires SLM learning for simple tasks. The role of the SLM is to organize the information from the vision module into one, and simple training is enough.

The simplicity of training only SLMs allows for faster learning with fewer datasets. Additionally, by using high-level vision features, information can be recognized more clearly. For example, it can better perform tasks like counting objects, which is a weakness of existing models. Another advantage of the EXAONE LMM model’s structure is that it is easy to add other vision modules such as chart recognition or OCR in the future to create task-specific models.

What Is Image Generation?

Image generation is also a task that generates images as answers to various input prompts from users based on its understanding of images. This is a field in which a lot of research is being conducted, starting with OpenAI’s Dall-E^[5] in 2021, Stability AI’s Stable Diffusion^[6], and Google’s Imagen^[7]. LG AI Research has also developed our own image generation model, which, along with image understanding technology, is being utilized in the design process of LG affiliates such as LG H&H as the name EXAONE Atelier. Similar to other models, it utilizes the basic algorithm, Denoising Diffusion Probabilistic Models (DDPM)^[8], and was developed into LG AI Research’s unique research. The generative model utilizing DDPM is described in detail in our previous post.

▶ [NeurIPS 2021] Part 1: Generative model - Diffusion model Review

Development of a Customize Image Generation Model

Before starting the New Hwadamsup project, we first tested how well the current EXAONE image generation model could represent Hwadam botanic garden. No matter how many prompts were entered to represent Hwadam botanic garden, not a single image that represented it could be generated, and the same was true for other image generation models. This is because the image generation models were not trained on Hwadam botanic garden's data. This makes sense, as it is the most data-dependent of all AI models. The goal of the New Hwadamsup project is to imagine a new Hwadam botanic garden that embodies the philosophy of the garden. We needed to develop a new customized model that could capture the philosophy of Hwadam botanic garden in the EXAONE Image Generation model.

First, we collected the needed data from Hwadam botanic garden for training. To train the image generation model, we need image-text pair data, but the only data that could be obtained was a video of Hwadam botanic garden. We used the EXAONE Image Understanding model to generate text describing the video of Hwadam botanic garden and processed it into an image-text pair dataset.

We made numerous attempts to apply the philosophy of Hwadam botanic garden to the EXAONE Image Generation model using LoRA^[9], and Dreambooth^[10] which are tuning techniques for generation models, and developed a custom model that contains the philosophy of Hwadam botanic garden.

Image 4. The New Hwadamsup Media Art Process

EXAONE Becomes the Hwadamsup Director

So then, how did EXAONE LMM plan Hwadam botanic garden’s media art? First, it looked at real photos of each season at Hwadam botanic garden and selected representative photos. Then, based on these photos, EXAONE LMM generated the elements needed to plan the media art. For example, EXAONE directed the direction of the exhibition by suggesting the basic story to express the Hwadam botanic garden in autumn, what kind of painting style should be used, musical elements to represent the overall atmosphere, and animation effects to be used in the media art (Image 5).

Image 5. Examples of recommended autumn-themed elements at Hwadam Botanic Garden

The basic direction was planned according to the elements recommended by the EXAONE LMM, and through the EXAONE Image Generation model and the hands of experts, the final Hwadamsup media art was born (Image 6).

Image 6. The New Hwadamsup Media Art

The New Hwadamsup project demonstrates the great potential of AI as an assistant for humans. The possibilities for AI-human collaboration in both media art and other fields are endless. AI can help us be more creative and efficient in art, science, technology, and other fields, and it can help us overcome human limitations. In the future, AI will become more advanced and seep deeply into various industries and everyday life, and collaborations between humans and AI will be the driving force behind new innovations and achievements. LG AI Research will continue to pave the way for AI and humans to create greater value together through continuous research and development.

▶The Four Season at Hwadam Botanic Garden, Generated by an AI trained on Nature