How did AI depict spring?

“Have you ever imagined AI painting spring?” These images were created by LG's super-giant AI, turning the text Newly Sprouting Spring Scenery into a visual painting. It came with the sentence, “spreading the bright energy of spring to the world.”

Vision AI, which mimics human vision, focused in its early stages on image recognition. This refers to recognizing objects such as vehicles and traffic lights on a road. Last year, an interesting case of OpenAI was introduced that learns from hundreds of millions of image-text pairs and generates image from text — text2image. It was the advent of an AI model dubbed DALL-E. “Avocado-shaped chair” images showed up in many places including icons. Vision AI is now focusing on “generation”.

LG AI Research’s EXAONE Multi-modal model went a step further than DALL-E. Whereas DALL-E generated image from text through uni-directional learning, EXAONE learned using a bi-directional model that generates text from images;[1] it can generate images from text and also describe images with words.

Generating an image from the text Newly Sprouting Spring Scenery is not an easy task for even human. It’s different from a simple request to draw a dog or a cat. Many different things could be utilized to depict spring energy as an image, such as flowers or frogs. The resulting image must express the overall feeling of spring. Flowers on a grave or a sleeping frog could hardly be described as suitable.

Then how does the EXAONE Multi-modal model execute such a task? In this blog, we’ll take a look at the details of the process. The figure below is an overview.

[Figure 1] EXAONE Multi-modal Model Overview

The process starts with the data on the left. These images and the description (language) of the images are image-text paired data. LG AI Research acquired a total of 250 million pairs of data for AI learning.

 

VAE(Variational Autoencoder): well-organized latent space

[Figure 2] AugVAE

Let's first look at the image learning on the top. This part depicts how the AI model learns from image data. It is the Feature-Augmented Variational AutoEncoder (AugVAE) that performs the learning here.

DALL-E was able to produce various images thanks to Vector Quantized Variational Autoencoder (VQ-VAE)[2]. LG AI Research's AugVAE is an enhanced version of the VQ-VAE.

Let’s take a closer look at VQ-VAE, starting with autoencoder. An autoencoder is a neural network model that learns how to compress a given data distribution in a latent space. This model can be split into two parts: the encoder that performs the compression and the decoder which performs decompression.

[Figure 3] Autoencoder Structure (Wikimedia Commons)

In this case, the data is in the form of images. When the image (X) passes through the encoder, it is transformed into a vector in the latent space (z in the figure above). The decoder then restores the vector back to the image (X’). Compression, as can be guessed from its name, will reduce the size of the expression (z) in the latent space that passes through the encoder when compared to input data (X). VAE’s excellence can be determined by how similar the recreated version is to the original after compressing the original data (X) into the latent space and then decompressing it (X’).

The Variational Autoencoder (VAE)[3]is an autoencoder with a twist — an autoencoder that “well organized” the latent space (z) which has converted the learned images. The following image is a good representation of the differences between an autoencoder and a VAE. Each point here refers to a learned data point. For images, this can be distinctive patterns. We can see that the autoencoder has points with the same color scattered in all directions. The VAE, however, has organized these points based on their characteristics in latent space.

 

[FIgure 4] Autoencoder and VAE (출처: ML@Berkely Understanding VQ-VAE)

 

AugVAE: a vision model that remembers characteristics regardless of image size

We’ve taken a look at VAE so far. Let’s go back to the beginning. What does Vector Quantized (VQ) in VQ-VAE mean? This is the part shown in the visual codebook below.

[Figure 5] Vector Quantization

A visual codebook, to put it simply, is an organized list that shows what parts of actual images the vectors in the latent space actually match up with. There are elements like “petals,” “leaves,” “cups,” and “houses” as can be seen in the above figure. VQ-VAE has a component for this codebook after the encoder and before the decoder, recording the characteristics of an image (Figure 5). One thing to take note of in the below figure is that the codebook organized as an example with image units is easy for humans to differentiate. Actual codebooks may not have such easily identifiable image units.

[Figure 6] VQ-VAE Diagram[4]

 

AugVAE is differentiated technically in this VQ aspect. Unlike VQ-VAE, which executes vectorizing at both the encoder and the decoder, AugVAE places this codebook component multiple times during the process of compressing the image as shown in the figure below.

[Figure 7] AugVAE's Vector Quantizer[5]

AugVAE is further improved than VQ-VAE through this as image patterns can be remembered well regardless of its size. There are zebras of many sizes in the following figure. Zebras have the same stripe pattern regardless of their size. Because AugVAE learns such patterns with precision regardless of size, it can recognize image patterns of various sizes and make use of them during generation.

[Figure 8] AugVAE Strengths

To sum up, the EXAONE multi-modal model which utilizes AugVAE effectively learns the sectional characteristics of various objects from a vast number of images. Such effective learning can be used to generate even more detailed images.

strong>A fusion of language and image

The lower part of the figure (an overview) refers to the process of turning text data into lists based on units of meaning (codebooks).

[Figure 9] Language Part of the EXAONE Multi-Modal Model

BP-Encoder refers to a Byte Pair Encoder, which is a widely-used algorithm that compresses data by replacing repeated characters with a different character and so on. This is used to organize image explanations into units of meaning and create a codebook as with image vectors. The codebook here is also an easy-to-understand example with vocabulary units as with the VQ-VAE codebook; actual codebooks may not use word units like this. Codebooks are organized by units of textual meaning according to the goal of image generation.

The rightmost part of the overview image shows how the AI learns the way a visual codebook and text codebook are combined.

[Figure 10] EXAONE Multimodal Model Transformer

The transformer model here is the part that learns the combination of text and images. Large amounts of example pairs are used by the transformer to learn how to combine image components with text. For example, it learns how it can generate a human-recognizable image of a flower with the characteristics of petals, leaves, and stems combined when given the keyword “flower.” EXAONE's AugVAE learns the characteristics regardless of the size of an image patch, showing off its stronger performance.

 

Towards the best image generation model

Based on such a structure, the EXAONE Multi-modal model, which has completed its learning based on vast amounts of data, can effectively generate beautiful images even when requested rather abstract requests like Newly Sprouting Spring Scenery.

Technology similar to that shown here was used for the image generation by the AI human with the first-ever super-giant AI brain — Tilda — for its debut as a designer(Tilda blog link). It was a successful collaboration between human designer who posed questions (text) and AI that created images to provide inspiration for beautiful fashion design.

LG AI Research’s Vision Lab, which studies this generation model, will focus on increasing the efficiency of the model to shorten generation time and expand the range of applications. One of the roadblocks for images being used in real-life is the time it takes to generate them. The final image may be only one image, but searching for the most suitable image takes a process involving the generation and review of thousands to tens of thousands of images. This requires an advanced technology that will create more suitable images in a shorter time frame.

With such technological advancement, image generation will be able to see more use cases in various areas apart from ad generation and design pattern generation. LG AI Research plans to achieve this goal and reach an unrivaled position in the AI image generation and design field.

[Figure 11] Pop Art Style Images of Newly Sprouting Spring Scenery

▶Multi-modal model AI #EXAONE (Link)