[ECCV 2022] Grounding Visual Representations with Texts for Domain Generalization

In this work, we advocate for leveraging natural language supervision for the domain generalization task. We introduce two modules to ground visual representations with texts containing typical reasoning of humans. Visual and Textual Joint Embedder aligns visual representations with the pivot sentence embedding. Textual Explanation Generator generates explanations justifying the rationale behind its decision. Our method achieves state-of-the-art results both in the newly created CUB-DG dataset and DomainBed benchmarks!

Introduction and Motivation

Machine learning systems assume that in-samples (training) and out-of-samples (test) are independent and identically distributed. However, this assumption rarely holds in real-world scenarios where domain shift often occurs. For example, think about a car image classifier model trained with photo images taken in urban areas. People would want it to work always well. However, it may deteriorate for test images that are quite different from those used in the training. It may not work well for sketch images. It may also not work well for photo images taken in rural environments. This is often called the domain generalization (DG) problem. Training a model with all possible domains can help but it is expensive and practically impossible. Therefore, improving the DG performance of artificial intelligence models holds great importance in both academic and industry fields.

Figure 1. Our model grounds visual representations with texts containing typical reasoning of humans.

In this work, we propose a novel approach that grounds visual representations with explicit (verbalized) knowledge from humans about typical reasoning on visual cues (Figure 1). We argue that learning from natural language supervision has potential strength over image-only training approaches, especially for the DG task. In contrast to the vulnerability of machine learning models against domain shift, the human visual recognition system generalizes well across domains. Thus, we advocate for using explicit knowledge from humans and we train a model to better align with the thought process of humans via their textual explanations. It ultimately provides more semantically-rich information compared to standard crowd-sourced labeling for image classification.

Methods

We aim to solve the DG problem: i.e. we train a model on single or multiple source domains $\left\{S_1,S_2,\cdots \right\}=S$ and evaluate it on unseen target domains, $\left\{T_1,T_2,\cdots \right\}=T$. Our model is built upon Empirical Risk Minimization (ERM), which minimizes the sum of errors across different domains^[1]. Formally, we use a task-specific loss function ${ℒ}_{task}$ over a model parameter $\theta$ where we use the following cross-entropy loss as we focus on the classification problem: 

where y is the one-hot vector representing each label's class and $\hat{\mathbf{y}}$ is the softmax distribution produced from the visual feature x.

Figure 2. An overview of our proposed model.
Our model leverages the text modality by (1) Visual and Textual Joint Embedder and (2) Textual Explanation Generator.

A key component of the DG task is learning domain-invariant representations that help improve the model generalization. In this work, we advocate for leveraging cross-modality supervision with semantic cues. Specifically, we use the following two main modules to ground visual representations with texts containing class-discriminative and high-level semantic information (Figure 2). First, the Visual and Textual Joint Embedder encourages our visual encoder to produce a latent representation that is aligned with textual semantics in the joint embedding space. Second, the Textual Explanation Generator produces a class-discriminative sentence detailing how visual evidence is compatible with a class prediction. Note that both modules are only required during the training phase for grounding the visual encoder. Nevertheless, the latter can also be optionally used during the inference to obtain textual explanation and a class prediction.

Visual and Textual Joint Embedder

We use a sentence-level textual encoder, which takes a variable-length sentence and yields a fixed-size latent vector v. Given the pivot textual latent representations v, we optimize the visual encoder to produce representations x that align well with the corresponding pivot. Thus, our model needs to understand the textual justification from human annotators and map it into the image-text joint embedding space. We assume that such textual justification will often contain class-discriminative evidence reflecting visual semantic cues, thus our visual encoder can internalize knowledge from natural language supervision.

Specifically, we minimize the following loss function ${ℒ}_{align}$ based on $l_2$ distance between the projected visual and texture features:

where $f_{proj}$ and $g_{proj}$ are the projection layers for text and visual features, respectively. $\widetilde{y_l}$ is the softmax distribution produced from the projected visual feature $g_{proj}{\left(\mathit{x}\right)}_i$. Note that we use the second cross-entropy term to make the projected visual features more class-discriminative. It prevents collapsing into collapsing solutions, e.g., always projecting them to the same point.

Pre-trained (Supervised) Textual Encoder (PTE)
One way to obtain the pivot textual latent representation is via pre-trained language models. These pre-trained encoders can be adopted from off-the-shelf sentence-level textual encoders that are often pre-trained with a large-scale dataset. In this work, we adopt the widely used CLIP model, which can embed texts and images into the joint representation space^[2]. The text encoder of CLIP is a 63M-parameter Transformer. It was jointly trained with a Vision Transformer (ViT)-based image encoder to predict the pairing of texts and images. In this work, we mainly used the text encoder of the CLIP-ViT-B-32 model, but other pre-trained language models are also applicable.

Self-supervised Textual Encoder (STE)
Another way to obtain the pivot textual latent representation is via self-supervision. Since our textual explanation generator justifies the rationale behind the model in the natural language, we can use it as a self-supervised textual encoder. As we will explain in the next subsection, during the training, we iteratively sample a sentence from a long short-term memory (LSTM)-based explanation generator to compute its training loss. Therefore, it is an intuitive choice to use its last hidden states as a fixed-size latent vector v.

Textual Explanation Generator
Our textual explanation generator is similar to image captioning models based on an encoder-decoder framework. It contains a two-layer LSTM network that takes high-level features from the visual encoder as input and generates variable-length per-word softmax probabilities. The difference is that it is trained to explain the rationale behind the classifier, reflecting typical visual semantic cues. Since it needs the prediction outputs from the classifier as an input as well, we concatenate the category information with a projected visual feature $g_{proj}\left(\mathit{x}\right)$. The concatenated vector is then used to update the LSTM network for a textual explanation generation.  

We use the discriminative sentence generation loss function based on reinforcement learning so that a model learns to generate sentences that are more likely to be class-discriminative^[3]. Specifically, we minimize the expectation of the negative reward $-R\left(\tilde{o}\right)$ over the sampled sentences $\tilde{o}~p\left(o\overline{)I,C}\right)$. The probability distribution $p\left(o\overline{)I,C}\right)$ is the model's estimated conditional distribution over descriptions conditioned on the input image I and the category C. We use the reward function as $R\left(\tilde{o}\right)=p\left(C\overline{)\tilde{o}}\right)$, which is the per-class softmax probabilities over the category conditioned on the generated sentence. A more class-discriminative sentence receives a higher reward. Concretely, we minimize the following loss function ${ℒ}_{expl}$:


Loss function
To summarize, we train our entire model end-to-end by minimizing the following loss function $ℒ$:


where we use hyperparameters ${\lambda }_{align}$ and ${\lambda }_{expl}$ to control the strengths of each training objective term.

Caltech UCSD Birds – Domain Generalization Extension (CUB-DG) Dataset

No previous DG benchmarks provide viable natural language supervision. Thus, to thoroughly investigate the effectiveness of the cross-modality supervision in the DG task, we have created a new benchmark built upon the CUB dataset^[4]. Since the CUB dataset is only composed of the Photo domain, we used pre-trained style transfer models to obtain images from three other domains, i.e. Art, Paint, and Cartoon^[5~7]. 

Figure 3. We created the CUB-DG dataset for investigating the effectiveness of natural language supervision in the DG task.

The generated CUB-DG dataset contains 11,768 sets of images and corresponding text descriptions. Each set illustrates the same content of a bird in four different domains (Figure 3). We train DG algorithms using non-overlapping groups from each source domain so that the different domains do not share the siblings of the same image. Then, we evaluate then on the test set from unseen target domains. Note that the CUB-DG dataset holds evident domain shifts such that an ERM model trained only on the Photo domain performs well on the same domain but significantly deteriorates on the other domains.

Experiment Results

Here, we focus on the multi-source DG task, where a single domain is used as a test domain and the others as training domains in rotation. We evaluate our model on the newly created CUB-DG dataset. We observe that our proposed models outperform the other approaches in all test domains (Table 1). While the performance difference between our model variants is marginal, we also observe that our model with the PTE generally shows better performance than a model with the STE.  

Since we only present our main results in this blog article, please refer to our paper(Link) for more extensive experiment results. They include several interesting observations of our proposed models. For example, the Joint Embedder plays a more crucial role in grounding visual representations with natural language supervision. We also compare the impact of embeddings from various PTEs. While we use CLIP as default, different PTEs also successfully produce domain-invariant representations.

Table 1. Multi-source DG evaluation results on the CUB-DG benchmark.

Next, we evaluate our model on the DomainBed benchmarks, which is a unified testbed useful for evaluating DG algorithms^[8]. While we have assumed Per-Image texts are available, obtaining them across different domains may not be easy in real life. Therefore, we introduce a more practical scenario where we only use the same single sentence for all the images within each class. Specifically, we use Per-Class textual definitions from Oxford English Dictionary^[9] to ground visual representations. The experiment results that the proposed algorithm shows state-of-the-art performance, where it ranks 1st in average performance for five multi-domain datasets. (Table 2).

Table 2. Multi-source DG evaluation results on the DomainBed benchmarks.

Conclusion

Towards learning more domain-invariant representations, we advocate for leveraging cross-modality supervision. Specifically, we propose a new approach where class-discriminative natural language sentence is used during training. Visual and Textual Joint Embedder encourages learning visual representations that align with the pivot sentence embedding. Textual Explanation Generator encourages to consistently verbalize why it made a certain prediction with natural language. The experiments with the newly created CUB-DG dataset and the DomainBed benchmarks show that our model outperforms prior work under the standard DG evaluation setting. Our analysis further shows that the text modality can be successfully used to justify visual predictions as well as improve the model's representational generalization power.

▶Grounding Visual Representations with Texts for Domain Generalization

- Paper (Link)
- Code (Link)