[CVPR 2022] Large Vision-Language Model: What’s Next?

Since the introduction of OpenAI GPT-3^[1] in 2020, the biggest topic of interest in the NLP sector has been large-scale model training that secures more data and infrastructure to learn larger models. The size of model is determined by the number of parameters of a model. As each parameter in a model plays the role of a storage space, generally speaking, the more the parameters, the wider the variety of information it can learn. But the larger the size of the model, the more the data and GPU are needed for learning, not to mention the probability of success decreases as well. That is why the size of the Large Language Model (LLM) disclosed by each research group acts as a scale showing the technologies of the said group.

Phase 1: Zero-shot text-to-image generation

In January 2022, when the ripple effects of GPT-3 were still in place, OpenAI unveiled another model through its blog. The model was called DALL-E^[2], which generates images from given texts. DALL-E greatly differed from the existing models in the text-to-image generation sector in terms of the model size and the amount of data used for learning. As a transformer model with 120 billion parameters, it learned 250 million images and pairs of text.

(Figure 1) Image generated by DALL-E^[2]. Chairs shaped like avocados (left) and kettles shaped like strawberries (right)

Furthermore, unlike the previous models that could only generate images similar to data used in learning, zero-shot text-to-image generation —generates images that were not learned such as “chairs shaped like avocados” or “kettles shaped like strawberries” (Fig. 1) — is possible. In order to enhance the accuracy of image generation according to the input text, DALL-E generates a total of 512 images for a single text, among which the image with the highest similarity with the text is selected. In the similarity analysis between image and text, CLIP^[3], which is a model unveiled together by OpenAI, was used.

(Figure 2) Overview of CLIP^[3]. Based on Contrastive Learning, CLIP can be used in various works that compare similarity between images and texts.

CLIP is also a model that learns 400 million images and the corresponding pairs of text. (Fig. 2) By using CLIP, which is a model that learns by pairing images and texts, it is possible to show the similarity between images and texts. The advent of DALL-E and CLIP, which learns by mobilizing massive amounts of data and computing infrastructure, brought the large-scale model training that was normally carried out in the NLP sector to the vision-language sector.

Phase 2: VQ-VAE (Vector Quantized-Variational AutoEncoder) + Transformer

DALL-E^[2] is based on GPT-3^[1], which is the most common LLM. As a decoder-only transformer, GPT-3 predicts words one by one that would come after an entered sentence. DALL-E additionally uses VQ-VAE^[4] to operate this model, which was used only for texts, and for images as well.

(Figure 3) Image compression and restoration process using VQ-VAE. The image compressed while passing the Encoder (left) goes through the Vector Quantization process (middle) and is sent to the Decoder (right); the Decoder returns the compressed image to its original size.

VQ-VAE is a type of VAE composed of an encoder that compresses images and decoder. The difference between VAE and VQ-VAE is that Vector Quantization (VQ) is additionally applied to information compressed by the encoder. VQ is a process of splitting information compressed by the encoder into tokens with specific sizes. It then replaces each of the tokens stored in the codebook with the most similar token. (Fig. 3) Figuratively speaking, it is similar to the process of splitting sentences drafted in different handwritings into words (tokens), and comparing each word (token) with words listed in the codebook to print it in a uniform font.

In summary, DALL-E is a model that uses VQ-VAE to change images into text-like formats and places them into the transformer with texts for learning. Though it is different from GPT-3 in detail, the structure is similar to GPT-3 which represents LLM in the NLP sector. Studies that were carried out afterward mainly made improvements under the large framework of VQ-VAE + Transformer.

In 2021, starting with CogView^[5] that uses 10% of learning data and 30% of parameters compared to DALL-E, various research results that outperform DALL-E with smaller amounts of data and less number of parameters begin to appear. Related research can also be found in 2022 CVPR, which is the top international academic conference in the field of computer vision; and L-Verse^[6] presented by LG AI Research is one of them.

(Figure 4) L-Verse of LG AI Research will be unveiled at CVPR 2022. Unlike the existing models like DALL-E, it can perform both text-to-image and image-to-text.

The biggest feature of L-Verse that will be presented in the oral session (Oral 4.1.3) on June 24 is that both text-to-image and image-to-text generation are possible (Fig. 4). Among them, image-to-text generation can be used for image captioning that explains input images. L-Verse can learn with 0.2% of learning data and 5% of parameters compared to DALL-E. It achieved outstanding performance compared to existing models in both the text-to-image and image-to-text sectors. For details on L-Verse, please refer to the previous posting and paper.

Phase 3: DDPM^[7]+ CLIP^[3]

Though the CVPR 2022 has not yet even begun, research on the Large Vision-Language Model (LVLM) that began with DALL-E has already entered into a new phase. The existing VQ-VAE + Transformer type has the advantage that learning is possible without having to make significant changes to the model structure. However, the loss of information that occurs in the course of changing 2D images made up of pixels into sentences cannot be avoided. Another limitation of DALL-E is that it cannot reflect correlation between the image and text learned by CLIP directly into the generation process and that it is only used for similarity calculations.

In late 2021, OpenAI presented the new LVLM called GLIDE^[8]. GLIDE uses the Denoising Diffusion Probabilistic Model (DDPM) instead of transformer for creating images. Being the latest one among the generative models, its operating principle is quite unique.

(Figure 5) Example of Diffusion Process. When a certain level of noise is repeatedly applied to an image, the image will only have noise in the end. The Denoising Diffusion Probabilistic Model (DDPM) ^[7] model learns the process of returning this. (Source)

DDPM can be explained as simply as possible without using any complicated equations as follows. When applying a certain amount of noise repeatedly to a clean image (Fig. 5), there will not be a big difference from the original image at first. But as it is applied increasingly, it will begin to lose its form, and as a result, the entire image will be filled with noise. This process of repeatedly entering a certain amount of noise is called the diffusion process. DDPM is a model that learns the method of carrying out the diffusion process in the reverse direction. It presumes that an image filled with noise has lost its original information due to the diffusion process, and it learns the denoising process through several steps.

The advantage of DDPM is that it can provide direct guidance on the step-wise denoising process. GLIDE brings in CLIP embedding corresponding to the text received by combining DDPM and CLIP to use as guidance. As result, unlike DALL-E that generated 512 samples and chose from one of them, GLIDE can generate images that properly reflect information in texts with a single sampling process. Such DDPM + CLIP structure was also applied in DALL-E 2.

Show and Tell: What’s Next?

It has already been 1 year and 6 months since DALL-E was released. Though CVPR 2022, an international academic conference for computer vision, has not yet even begun, the speed of development for the LVLM appears to have already surpassed the conference schedule.

When considering DALL-E 2^[9] OpenAI most recently unveiled and Google’s Imagen^[10], the current mainstream LVLM is DDPM + CLIP (text encoder). However, when taking into account the scalability of the transformer that is starting to be in the limelight in the NLP sector, it is still too early to come to a conclusion.

Unlike LLM research that learns human language by expanding the number of parameters to 175 billion, 300 billion, and 600 billion, current research on LVLM is focused on finding a structure for properly learning the relationship between images and texts. Whereas the limitations on the existing VQ-VAE + Transformer structure were discovered through Phase 1 and 2, Phase 3 is now at a stage where it is possible to utilize the DDPM optimized for image generation to create images similar to that drawn by people.

(Figure 6) Comparison of Google’s Imagen^[10] (left) and OpenAI's DALL-E 2^[9] (right). Compared to DALL-E2, Imagen generates letters with entered texts better.

However, when looking at the outputs of DALL-E 2 revealed through various media, while the generated images are realistic, the longer the entered texts, the less the contents are reflected. In particular, it displays weaknesses in generating letters (Fig. 6). This can be viewed to be due to the CLIP’s text encoder. In order to resolve this, Imagen^[10], which is the new LVLM presented by Google recently, uses an encoder-decoder type LLM — T5^[11] encoder — instead of CLIP.

Through this, Imagen can process longer input texts compared to DALL-E 2. It can also generate images including texts like “Store written as Text to Image” (Fig. 6). In addition, research using a transformer is ongoing. Recently, the Cogview structure was modified to present Cogview 2^[12] which displays performance similar to that of the DDPM + CLIP-based model. Deepmind unveiled Flamingo^[13] combined an image encoder to Chinchilla, which is a 70 billion parameter model based on the transformer. What type of Large Vision-Language Model (LVLM) will appear in Phase 4? Take a glimpse of Phase 4 at the CVPR 2022 from June 19 to 24.

▶L-Verse: Bidirectional Generation Between Image and Text (Link)
▶L-Verse: Bidirectional Generation Between Image and Text - YouTube (Link)