[NeurIPS 2022] Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching

CLIP (Contrastive Language-Image Pre-training)^[9] is a vision and language model that is pre-trained on large-scale image-text paired data collected from the Internet. As it provides surprisingly good zero-shot accuracy on previously unseen data samples, it can be generally used for diverse downstream tasks. For example, CLIP can recognize unseen images in the ImageNet test dataset with accuracy similar to that of a model trained on about 1.2 million labeled images in the ImageNet training dataset.

However, training such a model requires about 400 million image-text pairs collected from the Internet as well as hundreds of GPUs. Therefore, it is usually prohibitive in situations where time and computational resources are limited.

To address these limitations, we introduce BeamCLIP, a method that transfers the representations of a large pre-trained multimodal model such as CLIP, to a small target model such as ResNet-18. To enable BeamCLIP, we propose two methods: Cross-modal Similarity Matching (CSM) and Context-based Prompt Augmentation (CPA).

Figure 1 shows a comparison of the ImageNet top-1 linear probe accuracy on ResNet-18. Linear probing is an evaluation method used to assess the quality of deep representations after the neural network is trained. When used in downstream tasks, representations with higher linear probe accuracy usually result in better downstream accuracy. While vision-based Self-Supervised Learning (SSL) methods such as SimCLR^[2] and SwAV^[1] demonstrate 51.8% and 63.7% accuracy, respectively, BeamCLIP achieves a linear probe accuracy of 66.2%.. This suggests that BeamCLIP learns representations that are superior to those from vision-based SSL methods.

Conceptually, BeamCLIP can be seen as an unsupervised representation transfer method. For each unlabeled query image, the student model is trained to regress the vector representation of the teacher model. For vector regression, one typically uses L2 distance or cosine similarity. To alleviate the topological ambiguity that may occur in the similarity-based vector regression, we propose Cross-modal Similarity Matching (CSM). In addition, we propose Context-based Prompt Augmentation (CPA) to properly encode text prompts without lexical ambiguity. We will explain these methods in more detail in the following sections.

Figure 2 shows the overview of BeamCLIP. BeamCLIP trains the student model by using CLIP^[9] as the teacher model while freezing its parameters. CLIP consists of an image encoder and a text encoder, both of which are pre-trained, and the representations of each encoder are well-aligned in a cross-modal embedding space.

Cross-modal Similarity Matching (CSM)

In essence, BeamCLIP uses instance similarity matching (ISM) to train the student model to regress the vector representation from the teacher model. (Refer to the formula in Figure 3)

${\mathcal{L}}_{\mathrm{ISM}}=-\sum _{i=1}^N \left(\frac{f_{\theta }^S\left({\widehat{x}}_i\right)}{∥f_{\theta }^S\left({\widehat{x}}_i\right)∥2}\cdot \frac{f_{\varphi }^T\left({\widehat{x}}_i\right)}{∥f_{\varphi }^T\left({\widehat{x}}_i\right)∥2}\right)=-\sum _{i=1}^N \left(q_i\cdot k_i\right)$

Figure 3. Instance Similarity Matching (ISM) loss^[7]

Figure 4 illustrates the concept of Cross-modal Similarity Matching (CSM). When a query image $\left(x_i\right)$ is given, the vector representation $\left(q_i\right)$ of the student model $\left(f_{\theta }^S\right)$ is trained towards increasing its cosine similarity with the vector representation $\left(k_i\right)$ of the teacher model $\left(f_{\varphi }^T\right)$. However, instance similarity matching suffers from topological ambiguity, failing to optimize vector representations sharing the same angle but in different directions. In order to address this problem of topological ambiguity, BeamCLIP also optimizes cosine similarity with the vector representation of text prompts.

For example, in the teacher model’s embedding space, a dog image has high cosine similarity with the text prompt “a photo of a dog”, while it has low cosine similarity with the text prompt “a photo of a cat.” BeamCLIP trains the student model to maintain the cross-modal similarity of the teacher model.

Here, we explain the cross-modal similarity matching in more detail. First, we measure the relative relationship between images and texts in the teacher’s embedding space as follows. Given a query image, the normalized cross-modal similarity for a set of text prompts is measured. (Refer to the formula in Figure 5) Using this normalized cross-modal similarity, we then compute the cross-similarity distribution for a given set of the text prompts. (Refer to the formula in Figure 6)

Second, the student model is trained to attain the same cross-modal similarity distribution as the teacher model. Here, cross-entropy is used to match the cross-modal similarity distribution between the student and the teacher model. (Refer to Figure 7)

$s_j\left(k_i,A\right)=\frac{\exp \left(\left(k_i·a_j\right)/\tau \right)}{{\sum }_{m=1}^M\exp \left(\left(k_i·a_m\right)/\tau \right)}$

Figure 5. Cross-modal similarity^[7]

$P\left(k_i\mid A\right)=\left[s_1\left(k_i,A\right),\dots ,s_M\left(k_i,A\right)\right]$

${\mathcal{L}}_{\mathrm{CSM}}=\sum _{i=1}^{N}H\left(P\left(k_i\mid A\right),P\left(q_i\mid A\right)\right)+\sum _{i=1}^{N}H\left(P\left(q_i\mid A\right)\right)$

Context-based Prompt Augmentation(CPA)

Figure 8 illustrates the concept of Context-based Prompt Augmentation (CPA). To learn the teacher representation more accurately, BeamCLIP uses vector representations of text prompts as anchors. Because of the lexical ambiguity that can exist in a text prompt, however, a semantically inaccurate vector representation can be learned.

For example, the Flowers102^[8] dataset has a class name called “snapdragon.” However, it is ambiguous whether this refers to a flower or an animal. In this case, a vector representation of the text prompt “a photo of a snapdragon” may have a higher cosine similarity with “a photo of a dragon” than with “a photo of a flower”. To alleviate such lexical ambiguity, context-based prompt augmentation (CPA) augments the basic text prompts with Wikipedia descriptions or hierarchical labels.

Experiments
Comparison vs. vision-based Self-Supervised Learning (SSL) methods

Figure 9. Imagenet-1K top-1 linear probe accuracy on ResNet-18^[7]

Figure 9 shows the ImageNet-1K top-1 linear probe evaluation on ResNet-18. Achieving 63.8% accuracy after training for 100 epochs, BeamCLIP shows that it can learn better representations than SimCLR^[2] (51.8%), BYOL^[4] (61.4%), and SwAV^[1] (63.7%), all of which are vision-based self-supervised learning methods. It is worth noting that this linear probe accuracy of deep representations is achieved with relatively small neural networks such as ResNet-18, as self-supervised learning is known to not work well with small models. BeamCLIP demonstrates that, by using a pre-trained multimodal model such as CLIP^[9] as a teacher model, it is possible to learn good representations even with small models like ResNet-18.

Zero-shot performance comparison

Figure 10 shows the results comparing zero-shot performance on the ImageNet-1K^[3] dataset. On the ImageNet dataset, BeamCLIP-RN50 achieves a zero-shot accuracy of 57.5%. This is comparable to CLIP-RN50 which shows a ImageNet-1K zero-shot accuracy of 59.6%.

More importantly, CLIP^[9] requires about 400 million image-text pairs to achieve such zero-shot performance, while BeamCLIP only needs about 3% of the training data. This means that zero-shot performance comparable to that of CLIP can be obtained by using only about 12 million unlabeled images (that is, the datasets from ImageNet-21K excluding ImageNet-1K). OpenCLIP^[6], an open-source version of CLIP, only provides the accuracy of about 36.5% when trained on 12 million image-text pairs.

Comparison vs. existing Knowledge Distillation (KD) methods

Figure 11. Performance comparison of teacher and student models on various datasets^[7]

Figure 11 shows the performance comparison of student models trained with previous knowledge distillation methods on various datasets. It shows that both conventional Knowledge Distillation (KD)^[5] with KL-divergence and Contrastive Representation Distillation (CRD)^[10] with contrastive loss are ineffective for CLIP representation transfer. In contrast, BeamCLIP using Cross-Modal Similarity Matching (CMS) effectively transfers CLIP representations to ResNet-50 using unlabeled query images.

Qualitative analysis

Figure 12 shows a comparison of top-5 text-image retrieval results of CLIP-RN50 and BeamCLIP-RN50 on the Flowers102^[8] dataset. Images that do not match the given text prompt are denoted as red squares. As shown in the figure, CLIP-RN50 provides many inaccurate results. In contrast, BeamCLIP-RN50 shows improved text-image retrieval results, since its representations are transferred from CLIP-ViT that has higher zero-shot accuracy than CLIP-RN50. What is interesting is that BeamCLIP-RN50 can accurately retrieve similar images even for unseen text prompts such as “a photo of yellow rose”.

Conclusion

In this study, we introduce BeamCLIP, a method that can effectively transfer pre-trained multimodal representations such as CLIP^[9] to a small model such as ResNet-18. We empirically demonstrate that BeamCLIP can effectively transfer CLIP representations by using cross-modal similarity matching (CSM), which maintains the cross-modal similarity between image and text prompts, and context-based prompt augmentation (CPA), which extends text prompts to Wikipedia descriptions. By efficiently reusing high-quality representations of a large multimodal model that requires large training data and computing resources, BeamCLIP can reduce the cost and time required to develop multimodal AI applications. We hope this research leads to future progress on efficient machine learning frameworks with large-scale models which LG AI Research aims to develop.

▶ Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching (Link)