Pre-trained language models are still far from human performance in tasks that need understanding of properties (e.g. appearance, measurable quantity) and affordances of everyday objects in the real world since the text lacks such information due to reporting bias. In this work, we study whether integrating visual knowledge into a language model can fill the gap. We investigate two types of knowledge transfer: (1) text knowledge transfer using image captions that may contain enriched visual knowledge and (2) cross-modal knowledge transfer using both images and captions with vision-language training objectives. On 5 downstream tasks that may need visual knowledge to solve the problem, we perform extensive empirical comparisons over the presented objectives. Our experiments show that visual knowledge transfer can improve performance in both low-resource and fully supervised settings. 1 1 Introduction Pre-trained language models (PTLMs) such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and T5 (Raffel et al., 2020) have shown impressive results in various conventional natural language understanding (NLU) tasks by capturing syntactic and semantic knowledge from the pretraining tasks of masked language modeling and masked span infilling tasks on massive text corpora. Though yielding good performance on various NLU downstream tasks, these pre-training objectives suffer from a lack of out-of-domain knowledge that is not explicitly present in the pre-training corpus (Gururangan et al., 2020a; Petroni et al., 2021; Schick and Schtze, 2020). Specifically, one type of knowledge that models often struggle with is the visual knowledge of common objects such as attributes (e.g. appearance, measurable quantity)  Authors contributed equally. and affordances. This is because this kind of knowledge is rarely explicitly described in the training text due to reporting bias. For example, as shown in Figure 1, people tend to report what interests them rather than general facts such as a shape or color of oranges they already know. Towards better knowledge-enhanced PTLMs, re-cent works incorporate external knowledge bases (e.g., knowledge graph, dictionary) to inject entity knowledge into PTLMs (Zhang et al., 2019; Peters et al., 2019; Wang et al., 2021; Yu et al., 2021) or retrieve knowledge from external knowledge bases to solve the problem (Lin et al., 2019; Wang et al., 2020). However, these approaches still suffer from a lack of visual knowledge that is important to understand the real world. In this paper, we conduct systematic experiments to understand whether such visual knowledge can be transferred into LMs, and if so, how to perform effective knowledge transfer. Specifically, we look into a series of analysis question as follows: (1) Can intermediate pre-training (Pruksachatkun et al., 2020a) on image-caption pairs help transfer the knowledge? (2) What types of knowledge sources are more helpful? To answer questions, we explore various intermediate pre-training tasks (Pruksachatkun et al., 2020a) on two different sources: text-only ( text knowledge transfer from visual domains) and image-caption pairs ( cross-modal knowledge transfer ). corpus from visual domain, e.g., image captions. We leverage two training objectives for the language model: (1) masked language modeling follows the domain adaptive pre-training scheme (Gu-rurangan et al., 2020a), assuming the corpus contains enriched visual knowledge or physical commonsense knowledge; (2) text contrastive learning augments the sentence representation with dropout to create positive samples while considering all others in the batch as negative samples for the contrastive learning (Gao et al., 2021), assuming training better sentence representations leads to better understanding of the corpus. For the cross-modal knowledge transfer, we explore multiple methods to transfer visual-related knowledge to LMs: (1) masked language modeling with visual clues incorporates visual clues to capture dependencies between visual and linguistic contents (Su et al., 2020); (2) voken classification contextually aligns language tokens to their related images (called "vokens") to transfer visual knowledge into LMs (Tan and Bansal, 2020); (3) cross-modal contrastive learning aims to improve text representations by maximizing the agreement between correct image-text pairs versus random (in-batch) and adversarial negative pairs by contrastive learning between image and text modalities; and (4) cross-modal knowledge distillation transfers the knowledge from the teacher model, which is trained by cross-modal contrastive learning on image and text modalities, to the student language model using knowledge distillation. We perform comprehensive comparisons on five downstream tasks that may require visual or physical commonsense knowledge, including PIQA (Bisk et al., 2020), Visual Paraphrasing (VP) (Lin and Parikh, 2015), CSQA (Talmor et al., 2019), OBQA (Mihaylov et al., 2018), and RiddleSense (Lin et al., 2021). Results suggest that: (1) Simple intermediate pre-training on captions can help improving performance on commonsense reasoning that needs physical or visual knowledge. (2) Cross-modal knowledge transfer approaches consistently improve the performance in a large margin when only few train examples are available. (3) Cross-modal contrastive learning shows that it is best for packaging visual knowledge into LMs. In this work, we study how to transfer the visual knowledge into language models. For this study, we introduce our analysis setup: problem formulation, analysis questions, and knowledge corpora. We focus on a pre-trained text encoder f L and an image encoder f V if images are available. f L and f V are initialized with pre-trained model and we continue to pre-train the models on different sources and tasks, which we call intermediate pretraining (Gururangan et al., 2020b; Pruksachatkun et al., 2020b). After the intermediate pre-training, 2751 we fine-tune f L on downstream NLU tasks. Existing NLU benchmarks have been trained against standard supervised learning paradigms that typically require a large number of question answering examples which need a large annotation efforts. However, in scenarios where the number of labeled examples is small, the model tends to overfit the training examples and shows poor generalization performance on test set. Here, we evaluate the intermediate pre-training objective's generalization ability on test set in both fully supervised and low-resource settings. In this paper, we provide a comprehensive study for transferring the visual knowledge into LMs. Visual knowledge transfer can be done in two approaches, depending on the source to be trained: (1) Text knowledge transfer using the text corpus in the visual domain, e.g., image captions and (2) cross-modal knowledge transfer which passes visual knowledge about common objects to LMs by training over paired image and captions. By evaluating the model on 5 downstream datasets that require physical and visual commonsense knowledge, we explore following three research questions. Q1: Can intermediate pre-training on external knowledge sources help transfer visual knowledge to augment text encoders? We investigate diverse intermediate pre-training methods with external knowledge sources including caption data to inject visual information from images and captions into LMs. We first analyze the performance of text and cross-modal knowledge transfer methods with a image-caption dataset, and we additionally study text knowledge transfer methods with other text corpora such as GenericsKB (Bhakthavatsalam et al., 2020), Wiki103 (Merity et al., 2017) and BookCorpus (Zhu et al., 2015a). Q2: What types of knowledge sources are more helpful for visual knowledge transfer? As mentioned above, we have two categories to exploit visual information: (1) text knowledge transfer and (2) cross-modal knowledge transfer . Here, we explore which type of knowledge transfer is more useful to transfer the visual knowledge into LMs. Q3: What intermediate pre-training objectives are effective for cross-modal knowledge transfer? We present three pre-training objectives for cross-modal knowledge transfer: (1) voken classification, (2) contrastive learning, and (3) knowl-Dataset # Train # Dev # Test # choices PIQA 14,113 1,838 2,000 2 VP 21,988 2,000 6,057 2 CSQA 8,500 1,221 1,241 5 OBQA 4,957 500 500 4 RiddleSense 3,510 1,021 1,202 5 Table 1: Downstream task data statistics. edge distillation. Here, we want to present which strategy is best suited for cross-modal knowledge transfer. Furthermore, we study how to enhance cross-modal contrastive learning with adversarial negative samplings. To transfer the visual knowledge, we collect 250K image-caption pairs from MS COCO (Lin et al., 2014; Chen et al., 2015). MS COCO contains images reflecting the composition of actual everyday scenes and corresponding captions which describe contextual reasoning between objects in the scene. We only use captions for text knowledge transfer while we use both images and captions for cross-modal knowledge transfer. As an ablation study, we explore other text corpora such as GenericsKB (Bhakthavatsalam et al., 2020), Wiki103 (Mer-ity et al., 2017) and BookCorpus (Zhu et al., 2015a). For downstream benchmarks, we find tasks that can benefit from visual knowledge: multiple choice question answering tasks including PIQA (Bisk et al., 2020) which requires physical commonsense reasoning, CSQA (Talmor et al., 2019) for general understanding of commonsense reasoning, OBQA (Mihaylov et al., 2018) that needs elemenatry-level science knowledge, and RiddleSense (RS) (Lin et al., 2021) for complex understanding of figurative language, and binary classification task including Visual Paraphrasing (VP) (Lin and Parikh, 2015) that needs scene understanding. We use in-house test sets made from training sets for PIQA and CSQA since test set is not provided to public. We list the data statics in Table 1. Moreover, We additionally test on GLUE (Wang et al., 2019) to evaluate the general text understanding. We evaluate the models in both fully supervised and low-resource settings. For both settings, we consider accuracy for 5 different classification tasks and get average performance over tasks to check the final performance. In the fully supervised setting, we evaluate models with 3 different random seeds and report the average accuracy. In the low-resource setting, we set the size of the train data to 64 or 128. For each experiment, we run over 5 different sub-samples and show the average accuracy. In this section, we introduce the following two approaches to integrate visual knowledge into LMs: (1) text knowledge transfer ; and (2) cross-modal knowledge transfer . Throughout this section, we assume the data is a collection of image x v and caption x l pairs (cid:8) ( x v i , x l i ) (cid:9) m i =1 ( m is the size of the pairs) and image encoder f V and text encoder f L are given. Note that we use the same text encoder. For text knowledge transfer, we investigate following pre-training objectives: (1) masked language modeling ; and (2) text contrastive learning . Masked Language Modeling (MLM) Following BERT (Devlin et al., 2019), we select 15% of input tokens and replace them with [MASK] . Of the selected tokens, 80% are replaced, 10% are not changed and 10% are replaced by random vocabulary token. Here, we employ dynamic masking, which performs random masking and replacement during training to prevent the same masking for the same examples (Liu et al., 2019). MLM objective is the cross-entropy loss for masked token predictions :  MLM ( x li ) =  log p ( x li | x masked ) , (1) where x i is the i -th token and x masked is a mask. Text Contrastive Learning (TCL) Contrastive learning aims to learn representations by pulling positive pairs closer and pushing negative pairs apart. Here, we employ the contrastive framework with cross-entropy objective and in-batch negatives (Chen et al., 2020a; Gao et al., 2021). Given a text encoder f L , and a caption x li , we first get text representations using the encoders h li = f L ( x li ) . Following Gao et al. (2021), we create identical A girl puts an apple in her bag. positive sample h l + i by different dropout representations. The contrastive loss is defined as follows:  li =  log e sim ( h li ,h l + i ) / (cid:80) Nj =1 e sim ( h li ,h lj ) / , (2) where N is a batch size and sim (  ) represents co-sine similarity, i.e., sim ( u, v ) = u  v/  u  v  . Language models might learn additional information from visual sources such as images and captions. So we include a variety of vision-based approaches and investigate the approaches whether they can benefit from visual sources. We introduce vision-based approaches as follows. Voken Classification Vokenization (Tan and Bansal, 2020) employs token-level text-to-image retrieval to transfer visual knowledge. It aligns language tokens to their related images (called vo-kens) to transfer visual knowledge into LMs, and call it voken classification. Given text x and a voken v i for the i -th token, the loss is defined as  voken i =  log( p ( v i | x )) . Similar to masked language modeling, it classifies each token to a corresponding voken. Vokenization trains language models with the voken classification task and MLM. Masked Language Modeling with Visual Clues VL-BERT (Su et al., 2020) adopts masked language modeling with visual clues in which models are given a caption with masked tokens and an image and predict the masked tokens using visual clues. VL-BERT is pre-trained on Conceptual Captions (Sharma et al., 2018) as an image-caption corpus, and BooksCorpus (Zhu et al., 2015b) and English Wikipedia as text-only corpora. It shows 2753 its effectiveness in many vision-language tasks. We investigate whether this model also succeed in NLP tasks and compare it with others. Cross-modal Contrastive Learning (CMCL) To harness the visual knowledge from image-caption datasets, we adopt contrastive loss on image and text vectors. Given an image encoder f V , a text encoder f L , and an image-caption pair ( x vi , x li ) , we first get image and text representations using the encoders h vi = f V ( x vi ) , h li = f L ( x li ) . Then the contrastive learning objective contains two loss functions: an image-to-text contrastive loss  ( v,l ) and a text-to-image contrastive loss  ( l,v ) . The image-to-text contrastive loss is defined as follows:  ( v,l ) i =  log e sim ( h vi ,h li ) / (cid:80) Nj =1 e sim ( h vi ,h lj ) / , (4) where N is a batch size and sim (  ) represents co-sine similarity. This loss encourages a closer distance between representations of aligned image-caption pairs than unaligned pairs given an image and multiple captions. Similarly, the text-to-image contrastive loss  ( l,v ) is defined as follows:  ( l,v ) i =  log e sim ( h li ,h vi ) / (cid:80) Nj =1 e sim ( h li ,h vj ) / . (5) The final loss is defined as L = 1 NN (cid:88) i =1 (  ( v,l ) i +  ( l,v ) i ) . (6) CLIP (Radford et al., 2021) and ConVIRT (Zhang et al., 2020) also adopt contrastive learning, but we freeze the image encoder in training and use the trained text encoder for downstream tasks. CMCL with Adversarial Negative Samples (ANS) As in-batch negatives in CMCL are not challenging enough for models to distinguish, we present adversarial negative sampling strategy to improve CMCL. Given an image-caption pair ( x vi , x li ) , we define a LM-perturbed sentence x l  i , which is a hard negative where n is replaced with a different word n  from a probability distribution of PTLMs. We expect the l  is syntactically correct and plausible sentence even the word n is replaced to n  , while it does not semantically match to the corresponding image x vi . With such hard negative, we try to make more challenging task so that models can effectively learn from the task. For example, we choose a word girl' in the sentence A girl puts an apple in her bag.' in Figure 3. Then we mask the word with [MASK] token to do masked token predictions by PTLMs. Then we get top-k predictions from language models and replace the masked tokens with one of the predicted ones. To avoid false negative sentences which may have the same semantics as the original sentence, we introduce an additional filtering step: if the masked predictions are synonyms or hypernyms of the original tokens, we discard the predictions. We use WordNet (Miller, 1992) to find synonyms and hypernyms. The contrastive loss with hard negative is defined as follows:  log e sim ( h vi ,h li ) / (cid:80) Nj =1 e sim ( h vi ,h lj ) / + (cid:80) Mk =1 e sim ( h vi ,h l  j ) / , (7) where M is the number of hard negative samples per positive pair. This formula is only for image-to-text contrastive loss  ( v,l ) and final loss is defined to same as equation (6). CMCL with Positive Sample Augmentation (PSA) In ANS, we filter perturbed sentences where the masked predictions are synonyms or hypernyms of the original tokens. Instead of excluding these perturbed sentences, another option is to include them as additional positive samples l + to the paired images. We name this as positive sample augmentation (PSA). It also adopts LM-perturbed negative samples as in ANS. Cross-modal Knowledge Distillation (CMKD) Cross-modal knowledge distillation is to transfer knowledge between different modalities, e.g., image modality and text modality. In this category, CMKD is to transfer knowledge from a teacher model which is knowledgeable about visual information. VidLanKD (Tang et al., 2021) also utilizes a cross-modal knowledge distillation method to help with general language understanding. A teacher model is first trained using contrastive learning on a video-text dataset, and then it transfers its knowledge to a student language model using KD on a text corpus. Their contrastive learning loss (hinge loss) is defined as L = N (cid:88) i [max(0 ,   sim ( h vi , h li )+ sim ( h v  i , h li )) + max(0 ,   sim ( h vi , h li ) + sim ( h vi , h l  i ))] , (8) where v  and l  are a random image and caption text, respectively. is the margin between the similari-2754 Model PIQA VP CSQA OBQA RiddleSense Average 64 128 64 128 64 128 64 128 64 128 64 128 -BERT-base 52.6  0 . ties of a positive pair and a negative pair. Instead of video datasets, we use a MS COCO dataset to train a teacher model and use two versions of contrastive learning, equations (6) and (8). As another version of CMKD, we consider distilling visual knowledge from a pre-trained vision-language model, VL-BERT, which is knowledgeable about grounded language. We adopt masked language modeling on Wikitext103 (Merity et al., 2017), a subset of English Wikipedia, in the knowledge distillation step. For knowledge distillation, we adopt Neuron Selectivity Transfer (NST) (Huang and Wang, 2017), which proves the effectiveness in VidLanKD (Tang et al., 2021). For all the approaches, we use bert-base-uncased (Devlin et al., 2019) as text encoder f L and ResNeXt101 (Xie et al., 2017) as an image encoder f V . We continue to pre-train the encoders in our experiments. For text knowledge transfer, (1) MLM follows the exact setting of codebase in huggingface 2 which uses dynamic masking strategy to conduct language modeling task. (2) TCL conducts contrastive learning with f L . We choose the best checkpoint by the best spearman correlation on STSb (Cer et al., 2017). For cross-modal knowledge transfer, (1) CMKD explores VL-BERT, Vokenization, and VidLanKD approaches. Here, we use VL-BERT-large model to do CMKD. We use the VL-BERT and Vokenization checkpoints from their official 2 https://github.com/huggingface/transformers/ tree/master/examples/pytorch/language-modeling codebases 3 . VidLanKD trains a teacher model by two versions of contrastive learning (equations (6) and (8)) on MS COCO dataset. We set  = 1 in VidLanKD (equation (8)). (2) CMCL conducts contrastive learning with f L and f V . Here, we set  = 0 . 05 (equations (2) and (4)). (3) CMCL with ANS chooses three noun words or verb words to do masked prediction and use top-5 predictions from f L as replacement. We filter out synonyms and hypernyms of original words using WordNet (Miller, 1992). (4) CMCL with PSA includes the perturbed sentences with synonyms and hypernyms as additional positive samples. In CMCL, we adopt ResNeXt101 (Xie et al., 2017) as an image encoder f V and BERT as a text encoder f L . TCL and CMCL train with batch size 64, maximum sequence length 20, learning rate 1e-4 for 3 epochs. For fine-tuning on downstream tasks, we do grid search on learning rates {5e-5, 1e-4, 3e-4, 4e-4, 5e-4, 6e-4} and choose the best learning rate. We set maximum epochs to 30 in low-resource and 15 in fully supervised settings. We analyze the main results of intermediate pretraining. Tables 2 and 3 show the main results of low-resource learning and fully supervised learning with the MS COCO captioning dataset, respectively. We train the models with a few training examples, 64 and 128, to understand the better initialization. We argue that if a model obtains better performance in the low-resource setup, then it is a faster learner and has better generalization on downstream tasks. and second-best performance, respectively. text encoders? Text intermediate pre-training using MLM and TCL on a caption corpus improves the performance on downstream tasks in both low-resource and fully supervised settings. In particular, TCL shows significant improvement on OBQA and RiddleSense over BERT (p-value < 0 . 01 ). These results suggest that text intermediate pre-training on visual-related datasets helps performance on commonsense reasoning tasks. transfer visual knowledge to augment text encoders? We observe that cross-modal intermediate pre-training is helpful in both fully supervised and low-resource settings (See Table 2 and 3). Specifically, CMKD with VidLanKD variant outperforms the baseline by 1.6% point on the PIQA dataset in fully supervised setting. CMCL also shows its effectiveness. However, we could find that it becomes more powerful when equipped with PSA and ANS. It suggests that data augmentation for positive and negative sampling is an important factor for CMCL. In low-resource setting, we find that cross-modal knowledge transfer helps better initialization and lets models learn new tasks faster. What intermediate pre-training objectives are effective for cross-modal knowledge transfer? Among various cross-modal knowledge transfer methods, we study which method is the most effective for cross-modal knowledge transfer. Overall, CMCL with PSA and ANS shows the best performance among all cross-modal methods. Interestingly, VL-BERT also shows better performance than BERT-base on all datasets in the low-resource setting. This suggests that exploiting images in masked language modeling task help transfer the knowledge to language models. What types of knowledge sources are most helpful? Here, we investigate whether using an image source in addition to a text source can further improve the model. To answer this question, we analyze methods from different types of sources: text-only and text-image pair sources. We focus on the methods that use the contrastive learning objective: TCL and CMCL. Note that these two methods share the same objective but CMCL trains on cross modalities which are images and captions while TCL only trains on captions. Overall, TCL performs slightly better than CMCL in low-resource and fully supervised settings. Interestingly, additional negative samples (ANS) and positive samples in TCL decreases the performance while they help CMCL to improve the performance. We conjecture that perturbed sentences in ANS might not be semantically negative to the original sentence so models learn from wrong labels. How do models perform on general NLU tasks? Table 4 presents results on GLUE benchmark. In GLUE, text intermediate pre-training methods slightly underperform the original BERT-base. We conjecture that the intermediate pre-training on caption data might sacrifice knowledge of general language understanding. Analysis on diverse text corpora Table 5 represents text approaches with different pre-training corpora: MS COCO captions (Lin et al., 2014; Chen et al., 2015), GenericsKB (Bhakthavatsalam et al., 2020), BooksCorpus (Zhu et al., 2015a), and WikiText103 (Merity et al., 2017). We sample 250k sentences from each corpus for a fair comparison. We notice that caption datasets are useful on OBQA and RiddleSense datasets while GenericsKB are the most helpful on PIQA datasets. Results are expected since GenericsKB contains a lot of everyday 2756 0 500 1000 1500 2000 Training size 51 52 53 54 55 56 57 58 ACC o n PIQABERT CMCL CMCL+ANS CMCL+PSA+ANS Figure 4: Results on varying training sizes. Different training sizes. We test different training sizes on PIQA in Fig. 4. In the experiment, we observe that CMCL consistently outperforms BERT on all training sizes. Additional negative sample (ANS) improves the CMCL on different training sizes, and positive sample augmentation boosts the performance of CMCL further. This suggests including perturbed sentences as positive and negative samples are useful to cross-modal knowledge transfer. Text Knowledge enhanced methods. Recently, huge efforts on integrating knowledge into PTLMs have been made. One typical form of knowledge is a knowledge graph. There have been efforts of using knowledge graph to inject entity and relation representations, which are pre-computed from external source, into PTLMs (Zhang et al., 2019; Xu et al., 2021a; Peters et al., 2019; He et al., 2020; Xu et al., 2021b). Some other works try to retrieve or generate the sub-graph from the graph to solve the problem (Lin et al., 2019; Wang et al., 2020). Another existing form of knowledge is extra large-scale corpus. Works that use such corpus present knowledge-related pre-training objectives such as concept order recovering (Zhou et al., 2021), entity category prediction (Yu et al., 2020) and source of knowledge prediction (Wang et al., 2021; Calixto et al., 2021). They are mostly focused on injecting world knowledge presented in text, rather than physical and visual commonsense knowledge that can be found in images. Cross-modal knowledge enhanced methods. There is a extensive line of works for a variety of vision-language tasks, such as VL-BERT (Su et al., 2020), VisualBert (Li et al., 2019), and Uniter (Chen et al., 2020b). These models aim to improve vision-language tasks, e.g., VQA (Goyal et al., 2017) and event understanding (Li et al., 2022), and they are found to be not effective in improving language tasks (Tan and Bansal, 2020). Another line of works is to transfer visual knowledge to language models: Vokenization (Tan and Bansal, 2020) and VidLanKD (Tang et al., 2021). Vokenization employs token-level text-to-image retrieval to transfer visual knowledge to language models. For this, Vokenization introduces 30k vo-kens and matches each token into the limited voken space. VidLanKD adopts contrastive learning to train a teacher model on video datasets and uses distillation approaches to distill visual knowledge from the teacher to a student model. We study whether intermediate pre-training on visual knowledge can help transfer visual knowledge into LMs. We investigate text knowledge transfer and cross-modal knowledge transfer using images and captions. In our empirical analysis, we observe that intermediate pre-training on captions can help improving performance and cross-modal knowledge transfer approaches consistently improve performance. When the transfer methods are equipped with additional positive and negative samples, they show better performance. Future works include improving both commonsense reasoning and general language understanding.
abstain method objective objective result other abstain abstain abstain abstain abstain other abstain method method abstain abstain abstain other other method abstain abstain abstain method method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method abstain other method method method method method abstain abstain method method method method method abstain method abstain abstain method method method abstain other abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain method method method method method method method method method abstain abstain method abstain method abstain abstain abstain abstain abstain other abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other other other other other other other other other other other other other other other method objective result abstain abstain