[NeurIPS 2022] UniCLIP: Unified Framework for Contrastive Language-Image Pre-training

UniCLIP is a methodology that integrates contrastive learning that were independently used in different domains into a single framework to enhance data efficiency. For the integrated framework of contrastive learning, new problems were resolved in the three areas of architecture, contrastive loss, and similarity score, and the existing contrastive learning was further expanded. In addition, in image-text open dataset such as CC3M, CC12M, and YFCC15M, the effects of contrastive pre-training were verified and high performance was achieved compared to the existing CLIP method. 

Introduction

Contrastive learning used in the image unsupervised learning method SimCLR^[1] or the image-text pair unsupervised learning method CLIP^[2] have become very important technologies in representation learning. Contrastive learning defines positive pair and negative pair, and it is a method that makes embedding between positive pairs closer and embedding between negative pairs farther. Using the image and the corresponding text pair being positive pair and the image and the non-corresponding random text pair defined as negative pair, CLIP shows excellent image-text pre-training performance.

Figure 1. CLIP: Contrastive learning of image-text pairs^[2]

In papers such as SLIP^[3] or DeCLIP^[4], self-supervision loss was added in the CLIP for more efficient learning. But this methodology has the limitation that the contrastive loss between intra-domain pairs, such as image-image pairs, and inter-domain pairs, such as image-text pairs, are defined independently in a separate space. In this blog, we introduce the Unified framework for Contrastive Language-Image Pre-training (UniCLIP) technology that defined contrastive learning of all pairs in and between domains in a single integrated embedding space to overcome such limitations.

The biggest issue that occurs in the course of designing contrastive learning between multiple domains in a single embedding space is image-text misalignment due to augmentation. As shown in the Figure 3 below, it is evident that the relationship between image and text change completely just by basic image augmentations such as flip, grayscale, and crop. In order to solve such misalignment issue, a new multi-domain contrastive learning framework that can reflect augmentation information into embeddings is proposed in UniCLIP.

Figure 3. Image-text misalignment due to image augmentation^[5]

UniCLIP: Architecture

The biggest feature of the UniCLIP structure is the augmentation-aware projection head. Information on what type of augmentation was applied to the image is encoded through the augmentation encoder, and such augmentation information and image features pass through the projection head for final embedding. As the augmentation information can be known to the projection head in this structure, image-text misalignment issues caused by augmentation can be corrected by the projection head. 

Figure 4. UniCLIP structure^[5]

The Figure 5 below is an example of misalignment that can occur with flip augmentation. Since CLIP and SLIP do not use flip augmentation during learning, no problem is caused; however, DeCLIP uses flip augmentation during learning, and therefore, there are problems with misalignment. In contrast, UniCLIP does not suffer from the misalignment problem even though flip augmentations have used during training. And when augmentation information is provided to the projection head (the lowest line), it is evident that augmentation information is reflected for correction of misalignment.

Figure 5. Example of misalignment due to flip augmentation

Figure 6. Misalignment issues and performance due to augmentation^[5]

UniCLIP: Contrastive Loss

InfoNCE loss used in SimCLR or CLIP, has one positive sample (p_i) for every sample (i) as shown in the equation below.

 

Figure 7. InfoNCE loss

 

While there is no problem when independently calculating the contrastive loss per domain, but there may be more than one positive sample in UniCLIP that compares embedding of multiple domains in a single integrated space. To this end, UniCLIP proposes the Multi-Positive NCE(MP-NCE) loss, which is an extension of the InfoNCE loss that can be applied even with multiple positive pairs. 

Figure 8. MP-NCE loss

Figure 9. Performance of MP-NCE loss proposed in UniCLIP^[5]

The similarity score between two embeddings in existing contrastive learning is generally defined as follows.

Figure 10. Similarity score in contrastive learning

As the magnitude of cosine similarity cannot be greater than 1, in order to widen the range of the value that the score can have, temperature τ exists. But for situations where data are embedded in the same space from multiple domains like in UniCLIP, as each domain has different features, the appropriate scope of the score can also be different per domain. Based on this, UniCLIP proposes the following domain-dependent similarity score by defining the temperature and offset for each domain.

Figure 11. Domain-dependent similarity score proposed by UniCLIP

Through the domain-dependent similarity score, UniCLIP learns different appropriate temperature and offset for each domain D. As shown in the results of the Figure 12 below, it is evident that using domain-dependent temperature and offset will help with learning.

Figure 12. Performance of domain-dependent similarity score^[5]

Experiments

In the UniCLIP study, experiments were held on about 15 million pairs of image-text data YFCC15M and about 30 million pairs of image-text data Open30M (CC3M+CC12M+YFCC15M). Results showed that UniCLIP showed better zero-shot performance and linear probing performance in ImageNet, and Flickr30k and COCO Captions zero-shot image-text retrieval performance all exhibited better performance compared to the existing CLIP method.

Figure 13. ImageNet zero-shot performance and linear probing performance^[5]

Figure 14. Flickr30k and COCO Captions zero-shot image-text retrieval performance^[5]

Conclusion

UniCLIP is a contrastive learning framework that integrated contrastive learning used independently in SimCLR (image-image) and CLIP (image-text) into a single embedding space. For the contrastive learning in a single integrated embedding space, UniCLIP resolved new problems in the three areas of architecture, contrastive loss, and similarity score, and the existing contrastive learning was further expanded. Since the computational overhead for the parts added upon the existing methods is not large and it is easy to apply, it is assessed to have high usability. In particular, as UniCLIP can ease augmentation misalignment issues, the use of UniCLIP will become even more important in situations where stronger augmentation is needed because there is little data or for learning datasets vulnerable to augmentation misalignment.


▶UniCLIP: Unified Framework for Contrastive Language-Image Pre-training (Link)