This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextual multilingual multimodal embeddings. Under a zero-shot setting, we empirically demonstrate that performance degrades significantly when we query the multilingual text-video model with non-English sentences. To address this problem, we introduce a multilingual multimodal pre-training strategy, and collect a new multilingual instructional video dataset (Multi-HowTo100M) for pre-training. Experiments on VTT show that our method significantly improves video search in non-English languages without additional annotations. Furthermore, when multilingual annotations are available, our method outperforms recent baselines by a large margin in multilingual text-to-video search on VTT and VATEX; as well as in multilingual text-to-image search on Multi30K. Our model and Multi-HowTo100M is available at http://github.com/berniebear/ Multi-HT100M 1 Introduction One of the key challenges at the intersection of computer vision (CV) and natural language processing (NLP) is building versatile vision-language models that not only work in English, but in all of the world's approximately 7,000 languages. Since collecting and annotating task-specific parallel multimodal data in all languages is impractical, a framework that makes vision-language models generalize across languages is highly desirable. One technique that has shown promise to greatly improve the applicability of NLP models to new languages is zero-shot cross-lingual transfer , where models trained on a source language are applied  Equal contribution. as-is to a different language without any additional annotated training data (Tackstrom et al., 2012; Klementiev et al., 2012; Cotterell and Heigold, 2017; Chen et al., 2018; Neubig and Hu, 2018). In particular, recent techniques for cross-lingual transfer have demonstrated that by performing unsupervised learning of language or translation models on many languages, followed by downstream task fine-tuning using only English annotation, models can nonetheless generalize to a non-English language (Wu and Dredze, 2019a; Lample and Con-neau, 2019; Huang et al., 2019a; Artetxe et al., 2020; Hu et al., 2020). This success is attributed to the fact that many languages share a considerable amount of underlying vocabulary or structure. At the vocabulary level, languages often have words that stem from the same origin, for instance, desk in English and Tisch in German both come from the Latin discus. At the structural level, all languages have a recursive structure, and many share traits of morphology or word order. For cross-lingual transfer of vision-language models, the visual information is clearly an essential element. To this end, we make an important yet under-explored step to incorporate visual-textual relationships for improving multilingual models (De-vlin et al., 2019; Artetxe et al., 2020). While spoken languages could be different, all humans share similar vision systems, and many visual concepts can be understood universally (Sigurdsson et al., 2020; Zhang et al., 2020). For example, while is termed cat for an English speaker and chat for a French speaker; they understand similarly. We leverage this observation to learn to associate sentences in different languages with visual concepts for promoting cross-lingual transfer of vision-language models. In this work, we focus on multilingual text-to-video search tasks and propose a Transformer-based video-text model to learn contextual multilingual multimodal representations. Our vanilla model yields state-of-the-art performance in multilingual text  video search when trained with multilingual annotations. However, under the zero-shot setting, rather surprisingly, there is a significant performance gap between English and non-English queries (see 5.5 for details). To resolve this problem, motivated by recent advances in large-scale language model (Artetxe et al., 2020) and multimodal pre-training (Lu et al., 2019; Miech et al., 2019; Patrick et al., 2020), we propose a multilingual multimodal pre-training (MMP) strategy to exploit the weak supervision from large-scale multilingual text-video data. We construct the Multilingual-HowTo100M dataset, that extends the English HowTo100M (Miech et al., 2019) dataset to contain subtitles in 9 languages for 1.2 million instructional videos. Our method has two important benefits. First, compared to pre-training on English-video data only, pre-training on multilingual text-video data exploits the additional supervision from a variety of languages, and therefore, enhances the search performance on an individual language. Second, by exploiting the visual data as an implicit pivot at scale, our methods learns better alignments in the multilingual multimodal embedding space ( e . g ., cat--chat), which leads to improvement in zero-shot cross-lingual transfer ( e . g ., from cat-to chat) of vision-language models. In our experiments on VTT (Xu et al., 2016) and VATEX (Wang et al., 2019), our method yields state-of-the-art English  video search performance. For zero-shot cross-lingual transfer, the proposed multilingual multimodal pre-training improves English-video pre-training by 2  2 . 5 in average R@1 across 9 languages. Additionally, when trained with in-domain multilingual annotations as other baselines, our method outperforms them by a large margin in multilingual text  video search on VATEX and text  image search on Multi30K (El-liott et al., 2016). To summarize, we make the following contributions: (1) We propose a transformer-based videotext model that learns contextual multilingual multimodal representations (3.1). (2) We empirically demonstrate that vision-language models, unlike NLP models, have limited zero-shot cross-lingual transferrability. (5.5). (3) We introduce the multilingual multimodal pre-training strategy and construct a new Multi-HowTo100M dataset (4) for pre-training to improve zero-shot cross-lingual capability of vision-language models. (4) We demonstrate the effectiveness of our approach, by achieving state-of-the-art multilingual text  video search performance in both the zero-shot (5.5) and fully supervised setup (5.6). Cross-lingual representations. Early work on learning non-contextual cross-lingual representations used either parallel corpora (Gouws and Sgaard, 2015; Luong et al., 2015) or a bilingual dictionary to learn a transformation (Faruqui and Dyer, 2014; Mikolov et al., 2013). Later approaches reduced the amount of supervision using self-training (Artetxe et al., 2017). With the advances in monolingual transfer learning (McCann et al., 2017; Howard and Ruder, 2018; Peters et al., 2018; Devlin et al., 2019), multilingual extensions of pre-trained encoders have been proven effective in learning deep contextual cross-lingual representations (Eriguchi et al., 2017; Lample and Conneau, 2019; Wu and Dredze, 2019b; Siddhant et al., 2020; Pires et al., 2019; Pfeiffer et al., 2020). We extend prior work to incorporate visual context. Video-text representations. The HowTo100M dataset (Miech et al., 2019) has attracted significant interest in leveraging multimodal pre-training for text  video search (Korbar et al., 2020), captioning (Iashin and Rahtu, 2020), and unsupervised translation via image-based (Surs et al., 2020; Huang et al., 2020b) and video-based (Sig-urdsson et al., 2020) alignment. This work studies a challenging and unexplored task: Zero-shot cross-lingual transfer of vision-language models. Unlike prior image/video-text work that utilizes RNN (Dong et al., 2019; Chen et al., 2020a; Burns et al., 2020; Kim et al., 2020) and inter-modal contrastive objectives (Sigurdsson et al., 2020; Liu et al., 2019; Huang et al., 2019b; Patrick et al., 2021), we employ Transformers to learn contextual multilingual multimodal representations and uniquely models cross-lingual instances. Moreover, we build Multi-HowTo100M, the largest text-video dataset for multilingual multimodal pre-training. Cross-lingual Transfer. Cross-lingual transfer has proven effective in many NLP tasks including dependency parsing (Schuster et al., 2019), named entity recognition (Rahimi et al., 2019), sentiment analysis (Barnes et al., 2019), document classification (Schwenk and Li, 2018), and question an-mBERT 3D-CNN TP TP a man performs shot put un hombre realiza lanzamiento de bala A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n 1 D-CNN 1 D-CNN 1 D-CNN 1 D-CNN 3DCNN Time GRU a man shot put GRU GRU GRU  1 D-CNNA tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n L i n e a r 1 D-CNNA tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n L i n e a r A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n 1 D-CNN 1 D-CNN 1 D-CNN 1 D-CNN 3DCNN Time GRU a man shot put GRU GRU GRU  1 D-CNNA tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n L i n e a r 1 D-CNNA tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n L i n e a r A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n 1 D-CNN 1 D-CNN 1 D-CNN 1 D-CNN 3DCNN Time GRU a man shot put GRU GRU GRU  1 D-CNNA tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n L i n e a r 1 D-CNNA tt e n t i o n A tt e n t i o n A tt e n t i o n A tt e n t i o n L i n e a r time mBERT TP Contrastive attraction Contrastive repulsion Inter-modal Cross-lingual Intra-modal Transformer Pooling (TP)  ! We utilize intra-modal , inter-modal , and conditional cross-lingual contrastive objectives to align ( x, v, y ) where x and y are the captions or transcriptions in different languages of a video v . TP: Transformer pooling head. swering (Lewis et al., 2020; Artetxe et al., 2020). Recently, XTREME (Hu et al., 2020) was proposed to evaluate the cross-lingual transfer capabilities of multilingual representations across a diverse set of NLP tasks and languages. However, a comprehensive evaluation of multilingual multimodal models on zero-shot cross-lingual transfer capabilities is still missing. To our best knowledge, we are the first work that investigates and improves zero-shot cross-lingual transfer of vision-language models. We consider the problem of learning multilingual multimodal representations from a corpus C of video-text pairs { ( x i , v i ) } Ci =1 , where v i is a video clip and x i is its corresponding text (caption or transcription) that is written in one of K languages. Our goal is to learn a shared multilingual text encoder c x = ( x ) and a video encoder c v = ( v ) , both of which project the input to a shared D dimensional embedding space c v , c t  RD , where semantically similar instances ( i . e ., paired ( x i , v i ) ) are closer to each other than the dissimilar ones ( i . e ., ( x i , v j ) , i (cid:54) = j ). In the following, we denote a batch of multilingual text-video samples as B = { ( x i , v i ) } Bi =1 } where B  C . Figure 1 gives an overview of the proposed method. Our text encoder consists of a multilingual Transformer ( e . g . multilingual BERT (Devlin et al., 2019)) and a text Transformer pooling head (ex-plained below). Similarly, our video encoder consists of a 3D-CNN ( e . g . R(2+1)D network (Tran et al., 2018)) and a video Transformer pooling head. We use these multilingual multimodal Transformers to encode text and video for alignment. Unlike prior multilingual text-image models (Gella et al., 2017; Kim et al., 2020; Huang et al., 2019b) that utilize word embeddings and RNNs, our multilingual text encoder is built on a multilingual Transformer that generates contextual multilingual representations e x  RN  D to encode a sentence x containing N words. We employ an additional 2-layer Transformer which we will call a Transformer pooling head (TP) as it serves as a pooling function to selectively encode variable-length sentences and aligns them with the corresponding visual content. We use the first output token of the second Transformer layer as the final sentence representation. Precisely, we set c x = Trans (2) x ( query=key=value= e x )[0] where Trans (2) x is a 2-layer stack of Transformers (Vaswani et al., 2017) with e x as the (query,key,value) in the multihead attention. Note that we use the same text encoder to encode sentences in all languages. For encoding videos, our model uses pre-trained 3D-CNNs that encode spatial-temporal context in a video. For a M -second video v , we apply R(2+1)D (Tran et al., 2018) and S3D (Miech et al., 2020) networks to its frames, concatenate network outputs, and apply a linear layer to encode the visual input, e v  RM  D , to our model. Similarly to the text part, we employ a two-layer Transformer as the pooling head to encode videos with different lengths into fixed-length representations. Formally, we set c v = Trans (2) v ( query=key=value= e v )[0] . Since videos are typically long and have a high frame rate ( e . g ., 30 fps), it is infeasible to update 3D-CNNs simultaneously and therefore, we use pre-extracted video features. Our model is parameterized by  =  mBERT   Trans x   Trans v . For learning multimodal representations, the common practice is to minimize a contrastive objective to map the associated (video, text) embeddings to be near to each other in a shared embedding space. The inter-modal max-margin triplet loss has been widely studied in video-text (Yu et al., 2018; Liu et al., 2019) and image-text (Kim et al., 2020; Burns et al., 2020; Huang et al., 2019b) research. In this work, we generalize and model all inter-modal , intra-modal , and cross-lingual instances with a noise contrastive estimation objective (NCE) (Gut-mann and Hyvarinen, 2010; van den Oord et al., 2018; Chen et al., 2020b). Inter-modal NCE. Let X and V denote the subsets of the sampled sentences in multiple languages and videos in B , respectively. And let s ( a, b ) = a T b (cid:107) a (cid:107)(cid:107) b (cid:107) be the cosine similarity measure. We use an (inter-modal) NCE objective defined as: L ( X , V ) =  1 BB (cid:88) i =1 log (cid:96) NCE (( x i ) , ( v i )) , (1) where (cid:96) NCE ( c x , c v ) = e s ( c x ,c v ) e s ( c x ,c v ) + (cid:80) ( x (cid:48) ,v (cid:48) ) N e s ( c x (cid:48) ,c v (cid:48) ) (2) In inter-modal NCE, L inter = L ( X , V ) , the noise N is a set of negative video-text pairs sampled to enforce the similarity of paired ones are high and and those do not are low. Following Miech et al. (2020), we set the negatives of ( x i , v i ) as other x j and v j , j (cid:54) = i in B . Intuitively, inter-modal NCE draws paired (se-mantically similar) instances closer and pushes apart non-paired (dissimilar) instances. Note that we do not distinguish language types in X and the sentences in all possible languages will be drawn towards their corresponding videos in the shared multilingual text-video embedding space. Intra-modal NCE. Beyond cross-modality matching, we leverage the intra-modal contrastive objective to learn and preserve the underlying structure within the video and text modality. For example, Corgi should be closer to Husky than Balinese . Prior image-text work (Gella et al., 2017; Huang et al., 2019c) utilizes a triplet loss to maintain such neighborhood relationships. Inspired by recent success in self-supervised image and video representation learning (Yalniz et al., 2019; Ghadiyaram et al., 2019), our model leverages intra-modal NCE that constrains the learned representations to be invariant against noise and to maintain the within-modality structure simultaneously. We minimize the following intra-modal NCE loss: L intra = L ( X , X m ) + L ( V , V m ) , (3) where X m and V m are the noised version of the original sentences and videos. For noising, we randomly mask 5% of the multilingual text tokens and video clips. We optimize our model by min  L inter + L intra (4) 3.3 When Visually-Pivoted Multilingual Annotations Are Available In many multilingual multimodal datasets, there are sentences in different languages that describe a shared visual context. For example, 10 English and 10 Chinese descriptions are available for each video in VATEX. With these visually-pivoted (weakly paralleled) sentences ( x, y ) , we further revise the contrastive objectives to leverage this additional supervisory signal. Given a visually-pivoted corpus C p that contains all possible combination of visually-pivoted pairs { ( x i , v i , y i ) } C p i =0 , we sample batches B p = { ( x i , v i , y i ) } B p i =1 , B p  C p and revise the contrastive objective as: L inter = L ( X , V ) + L ( Y , V ) (5) L intra = L ( X , X m ) + L ( Y , Y m ) + L ( V , V m ) (6) Visual-pivoted Cross-lingual NCE. Inspired by Translation Language Modeling (TLM) in XLM (Lample and Conneau, 2019), we propose a multimodal TLM-like contrastive objective which promotes alignments of descriptions in different languages that describe the same video. We use the intuition that conditioned on a video, the descriptions (need not to be translation pairs) in different languages would likely be semantically similar. To this end, we set the cross-lingual NCE as: L cross = L ( X |V , Y|V ) (7) For visually-pivoted sentences, as shown in Fig. 1, we generate their representations conditioned on the video they describe. We extend the key and value of multihead attention with the additional visual content e v and generate new c x | v and c y | v for matching. Specifically, our model employs c x | v = Trans (2) x ( query= e x , key=value= e x || e v )[0] . With the access to (visually-pivoted) multilingual annotations, we optimize our model by min  L inter + L intra + L cross (8) yafries yaKifaransaunawezapia kuandamananayo tambinla voyaacompaarcon un poco de papas fritas und dann ziehen Sie es so fest wie mglich  It will also be accompanied with a little of frenchfries        What it is, is a heat gun and I got this for ten bucks 00:00:37.160 --> 00:00:48.860 we just made our six-sided coaster so and then pull it as tight as possible nous venonsde faire notrecaboteursix ctsdoncceque 00:11:36.380 --> 00:11:44.390 00:08:35.289 --> 00:08:39.300 von Pommes Frites knnenSie es auchmitbegleiten khoaitychinbncngcthikmvin 00:01:16.290 --> 00:01:21.210 Figure 2: Video clips and the corresponding multilingual subtitles in Multi-HowTo100M. At the inference time, we simply apply c x = ( x ) and c v = ( v ) to encode multilingual text queries and videos. For text-to-video search, we sort videos according to their cosine similarity scores to the text query. As large-scale pre-training has been shown important in recent NLP and vision-language models, we construct the Multilingual HowTo100M dataset (Multi-HowTo100M) to facilitate research in multilingual multimodal learning. The original HowTo100M (Miech et al., 2019) dataset is a large-scale video collection of 1.2 million instructional videos (around 138 million clips/segments) on YouTube, along with their automatic speech recognition (ASR) transcriptions as the subtitles. For each video in HowTo100M, we crawl and collect the multilingual subtitles provided by YouTube, which either consist of user-generated subtitles or those generated by Google ASR and Translate in the absence of user-generated ones. Essentially, we collect video subtitles in 9 languages: English ( en ), German ( de ), French ( fr ), Russian ( ru ), Spanish ( es ), Czech ( cz ), Swahili ( sw ), Chinese ( zh ), Vietnamese ( vi ). At the time of dataset collection (May 2020), there are 1.1 million videos available, each with subtitles in 7-9 languages. The video length ranges from 1 minute to more than 20 minutes. We utilize Multi-HowTo100M for multilingual multimodal pre-training to exploit the weak supervision from large-scale multilingual text-video data. In Fig. 2, we provide a visualization of few instances sampled in Multi-HowTo100M with the corresponding video frame, timestamp, and transcriptions in different languages. Please refer to Appendix for more details and dataset statistics. In this section, we first describe our experimental setup (5.1-5.3). In 5.4, we conduct ablation studies to validate the effectiveness of proposed multilingual text-video model . With the best models at hand, we investigate their zero-shot cross-lingual transferability in 5.5, where we showcase that the proposed multilingual multimodal pre-training serves as the key facilitator. We then verify the superior text  video search performance of our method under the monolingual, multilingual, and cross-modality settings in 5.6. MSR-VTT (VTT) (Xu et al., 2016) contains 10K videos, where each video is annotated with 20 captions. Additionally, we created pseudo-multilingual data by translating the English captions into 8 languages with off-the-shelf machine translation models. 1 We use the official training set (6.5K videos) and validation set (497 videos). We follow the protocol in Miech et al. (2019); Liu et al. (2019) which evaluates on text  video search with the 1K testing set defined by Yu et al. (2018). VATEX (Wang et al., 2019) is a multilingual (Chi-nese and English) video-text dataset with 35K videos. Five ( en , zh ) translation pairs and five non-paired en and zh descriptions are available for each video. We use the official training split (26K videos) and follow the testing protocol in Chen et al. (2020a) to split the validation set equally into 1.5K validation and 1.5K testing videos. Multi30K (Elliott et al., 2016) is a multilingual extension of Flickr30K (Young et al., 2014). For each image, there are two types of annotations available: (1) One parallel (English,German,French,Czech) translation pair and (2) five English and five Ger-1 https://marian-nmt.github.io/ man descriptions collected independently. The training, validation, and testing splits contain 29K, 1K, and 1K images respectively. For the video backbone, we use a 34-layer, R(2+1)-D (Tran et al., 2018) network pre-trained on IG65M (Ghadiyaram et al., 2019) and a S3D (Miech et al., 2020) network pre-trained on HowTo100M. We pre-extract video features and concatenate the two 3D-CNN outputs to form e x  RM  1024 as a video input. For the text backbone, we use multilingual BERT (mBERT) (Devlin et al., 2019) or XLM-Roberta-large (XLM-R) (Artetxe et al., 2020), where the latter achieves near SoTA zero-shot cross-lingual transfer performance for NLP tasks. Following Hu et al. (2020), instead of using the top layer, we output the 12-th layer in XLM-R and mBERT. For vision-language tasks, we freeze layers below 9 as this setup empirically performs the best. Our model employs a 2-layer Transformer with 4-head attention for the text and video transformer pooling (TP) modules. The embedding dimension D is set to 1024. We use the Adam (Kingma and Ba, 2015) optimizer and a 0 . 0002 learning rate to train our model for 16 (pre-training) and 10 (fine-tuning) epochs. The softmax temperature in all noise contrastive objectives is set to 0 . 1 . We use Multi-HowTo100M for multilingual multimodal pre-training (MMP). For each video, we randomly sample the start and end time to construct a video clip. For a video clip, we randomly sample one language type each time from 9 languages and use the consecutive ASR transcriptions that are closest in time to compose (text-video) pairs for training. For simplicity and speed purposes, we follow the training protocol of XLM-R to pre-train on a multilingual corpus wihtout using translation pairs, i . e ., we use multilingual text-video pairs ( x, v ) but no translation pairs from Multi-HowTo100M and utilize only interand intra-modal NCE (Eq. 1-3) for MMP. We fine-tune our model on VTT, VATEX, and Multi30K to evaluate on text  video search tasks. In the zero-shot cross-lingual transfer experiments, we use only English-video data and fine-tune with Eq. 1-3. We then test the model with non-English queries. When annotations in additional languages are available (by humans in VATEX and Multi30K; Text-B Video-B R@1  R@5  R@10  XLM-R S3D 19.5 49.0 62.8 XLM-R R(2+1)D 19.0 49.5 63.2 XLM-R R+S 21.0 50.6 63.6 mBERT R+S 19.9 49.8 62.5 Table 1: Text and Video (B)ackbone comparison. by MT models ( i . e ., translate-train ) in VTT), we utilize all available multilingual annotations ( i . e ., fully supervised) and iterate over all possible ( x, v, y ) pairs to train with Eq. 5-7 to demonstrate the strong performance target for evaluating zero-shot cross-lingual transfer on VTT and to compare fairly with other fully-supervised baselines in multilingual text  video search on VATEX and Multi30K. We report the standard recall at k (R@ k ) metrics (higher is better). In this section, we ablate and compare different text/video encoders, Transformer model architectures, and learning objectives for English  video search on VTT. Text and Video Encoders. Table 1 compares different text and video encoder backbones. For the visual encoders, while R(2+1)D outperforms S3D, the simple concatenation ( i . e ., early-fusion) of their output features provides a 1 . 5  2 . 0 improvement in R@1. For the text encoder, XLM-R significantly outperforms mBERT. Transformer Pooling. Table 2 compares various configurations of the proposed Transformer pooling module. We observe that a simple 2-layer Transformer achieves the best performance. Weight Model en de fr cs zh ru vi sw es Avg  mBERT 19.9 11.1 11.6 8.2 6.9 7.9 2.7 1.4 12.0 9.1 mBERT-MP 20.6 11.3 11.9 8.0 7.1 7.7 2.5 1.1 12.5 9.2 mBERT-MMP 21.8 15.0 15.8 11.2 8.4 11.0 3.7 3.4 15.1 11.7 XLM-R 21.0 16.3 17.4 16.0 14.9 15.4 7.7 5.7 17.3 14.7 XLM-R-MP 23.3 17.4 18.5 17.1 16.3 17.0 8.1 6.2 18.5 15.8 XLM-R-MMP 23.8 19.4 20.7 19.3 18.2 19.1 8.2 8.4 20.4 17.5 mBERT + translated VTT 19.6 18.2 18.0 16.9 16.2 16.5 8.4 13.0 18.5 16.1 mBERT-MMP + translated VTT 21.5 19.1 19.8 18.3 17.3 18.3 8.9 14.1 20.0 17.4 XLM-R + translated VTT 21.5 19.6 20.1 19.3 18.9 19.1 10.3 12.5 18.9 17.8 XLM-R-MMP + translated VTT 23.1 21.1 21.8 20.7 20.0 20.5 10.9 14.4 21.9 19.4 Table 4: Recall@1 of multilingual text  video search on VTT. Learning Objective. From Table 3, the intra-modal contrastive objective is important for both NCE and Triplet loss. In general, the NCE loss outperforms the Triplet loss. The proposed inter-modal and intra-modal NCE objective achieves the best performance. When captions in multiple languages are available, cross-lingual NCE additionally provides a consistent improvement. Table 4 shows the multilingual text  video search results on VTT. With the best English-video models at hand (with either mBERT or XLM-R as the text backbone), we first investigate how well these models transfer to other non-English languages under the zero-shot setting. We then analyze the benefit of the proposed multilingual multimodal pre-training. The upper section shows the zero-shot results. Unlike cross-lingual transfer in NLP tasks, employing multilingual Transformers in vision-language tasks apparently does not generalize well across languages. For example, there is a significant drop in R@1 (19.9  11.1 (-44%) with mBERT, 21.0  16.3 (-24%) with XLM-R) when directly applying English-finetuned model to German  video search. For comparison, there is only a -10% degradation for XLM-R on en  de cross-lingual transfer in XNLI (Conneau et al., 2018). Multimodal (English-video) pre-training (MP) on HowTo100M only improves average R@1 (+0.1 or mBERT and +1.1 for XLM-R) compared to model-from-scratch. In contrast, our proposed multilingual multimodal pre-training (MMP) is shown to be the key facilitator for zero-shot cross-lingual transfer. MMP improves German  Video search (11.1  15.0, +35% for mBERT, and 16.3  19.4, +20% for XLM-R) and achieves 2 . 6  2 . 8 improvement in average R@1. We attribute the effectiveness of MMP to learning improved alignments between multilingual textual and visual context in the shared embedding space, as relatively balanced improvements between English  video and non-English  video is observed with fine-tuning. Fig. 3 demonstrates the trend of R@1 while incrementally incorporating additional languages for MMP. For XLM-R, the improvement in R@1 asymptotically converges when pre-training with more multilingual text-video pairs. On the other hand, for zero-shot German  video search, pretraining with more languages keeps improving the search performance, even though the additional language ( e . g ., French) is different from the target language ( i . e ., German). The lower section of Table 4 shows the results of models fine-tuned with (synthesized) pseudo-multilingual annotations. It can be regarded as the translate-train scenario, which serves as a strong performance target for evaluating zero-shot cross-lingual transfer, as discussed in (Lample and Conneau, 2019; Hu et al., 2020). Both mBERT and XLM-R yield better performance across non-a soccer team walking out on the field 1 2 3 1 2 3 (0.69) (0.58) (0.53) (0.71) (0.47) (0.54)          Rank 1 2 3 (0.52) (0.48) mt ngi n ng ang ni v d n khng gian adam (0.44)  1 2 3 (0.45) (0.42)  (0.46)  Figure 4: Qualitative multilingual ( en , ru , vi , zh ) text  video search results on VTT. English languages with the in-domain translated pseudo-multilingual annotations. However, for English  video search, a 0 . 7 degradation is observed compared to the zero-shot setting. It is likely due to the noise in the translated captions. Notably, there is still a performance gap between zero-shot and translate-train settings for models with mBERT. In contrast, the gap is much smaller for models with XLM-R. In the following sections, we refer Ours-MMP as our best model with XLM-R as the text backbone and compare it with other state-of-the-art methods. Qualitative Results Fig. 4 shows the multilingual text  video search results with Ours-MMP (VTT: en -only) on VTT under the zero-shot setup. Note that only one shared English-finetuned model is used for text  video search in all languages. As demonstrated, the proposed model successfully retrieves the correct videos with English ( en ) and Russian ( ru ) queries. The other top-ranked videos also share similar visual appearance to the correct one. For zero-shot transferring of the English-finetuned model to distant languages such as Vietnamese ( vi ) and Chinese ( zh ), we observe that there is still limitation for our zero-shot models to understand abstract concepts ( e . g ., space project) and associate small objects ( e . g ., microphone) with the text queries in distant languages. English  Video Search on VTT. Table 5 shows the comparison of English  video models on VTT. For a fair comparison to other baselines, our model fine-tunes only with the original English annotations on VTT. The results show that our model outperforms other baselines by a large margin. Specifically, our model achieves 8.9 R@1 improvement over the original HowTo100M model (Miech et al., 2019) and other recent baselines with pre-training on HowTo100M. Using a smaller set of visual fea-Model R@1  R@5  R@10  JSFusion (Yu et al., 2018) 10 . tures and training on a smaller (6,513 vs 9,000) training set 2 , our model also outperforms CE (Liu et al., 2019) with or without pre-training. Multilingual Text  Video Search on VATEX. Table 6 summarizes English  video and Chinese  video search performance on the VATEX dataset. Under the zero-shot setting where we train with only English-video pairs, our model already outperforms other baselines. However, a clear performance gap between English  video and Chinese  video search is observed, indicating that cross-lingual transfer to a distant language remains challenging even with XLM-R. With the proposed MMP, the gap is significantly closed by 5.8/8.1/7.7 in R@1/5/10. When in-domain human-annotated Chinese captions are available, the performance of our model can further be improved for both languages and our model yields new state-of-the-art performance. 2 CE uses 9,000 videos (VTT training and part of exclusive testing set) for training, while other baselines and our model in Table 5 are trained on the official VTT training set which contains 6,513 videos. Cross-Modality Transfer to Multi30K: From Video-Text to Image-Text. To extend our study on zero-shot cross-lingual transfer for image-text tasks, we investigate the feasibility of transferring our video-text model across modalities. We replace the 3D-CNN in the original video-text model with a 2D-CNN to encode the image. In practice, following MHA-D (Huang et al., 2019b), we utilize the Faster-RCNN (Ren et al., 2015) pre-trained in Visual Genome (Krishna et al., 2016) to extract regional visual features. Essentially, an image is encoded as e v = RM  H where M = 36 is the maximum number of visual objects in an image. For models with MMP, we initialize their weights with the model pre-trained on Multi-HowTo100M. To tackle the feature mismatch between 2D-CNN and 3D-CNN, we leverage a linear layer with a doubled learning rate to map 2D-CNN features to the same dimension as 3D-CNN features. Table 7 shows the results on Multi30K. For zero-shot cross-lingual transfer, when trained from scratch (M30K: en -only), our model achieves comparable performance to MHA-D but lags in German  image search since it only uses English annotations. In Ours-MMP, pre-training improves all recall metrics even with modality gap. The average R@1 improvement is 3.2. A larger gain for (relatively) low-resource language such as Czech is observed. Without using any Czech annotations, our zero-shot model with MMP achieves comparable Czech  image search performance to SMALR (Burns et al., 2020), which uses 10 languages including Czech. However, when transferring across modalities and using only English annotations, there are performance gaps between English  Image and German/Czech  Image search, implying that transferring models across modalities is feasible but remains challenging. We consider zero-shot cross-modal cross-lingual transfer as our future work. For a fair comparison with other baselines, when trained with annotations in all 4 languages provided by Multi30K, our model greatly outperforms all baselines by large margins in multilingual text  image search. We have presented a multilingual multimodal pretraining (MMP) strategy, the Multi-HowTo100M dataset, and a Transformer-based text-video model for learning contextual multilingual multimodal representations. The results in this paper have convincingly demonstrated that MMP is an essential ingredient for zero-shot cross-lingual transfer of vision-language models. Meanwhile, there are many remaining challenges, such as resolving the performance gap between zero-shot and training with in-domain non-English annotations; as well as techniques to transfer varieties of vision-language models ( e . g ., VQA (Goyal et al., 2017), TVQA (Lei et al., 2020)) or visually-enhanced NLP models such as unsupervised multimodal machine translation (Huang et al., 2020b). We believe the proposed methodology, and the corresponding resources we release, will be an important first step towards spurring more research in this direction. This work is supported by the DARPA grants funded under the AIDA program (FA8750-18-2-0018) and the GAILA program (award HR00111990063) (P.Y.). This work is also supported by EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines & Systems [EP/L015897/1] (M.P.). The authors appreciate Prahal Arora, Shengxin Zha, Polina Kuznetsova, Xu Hu, and Geoffrey Zweig for their suggestions of this work. The authors would also like to thank the anonymous reviewers for their feedback.
abstain objective objective objective result result other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain abstain abstain abstain method abstain abstain abstain result objective objective abstain objective objective other other other other objective other other method other method other other objective other other other other objective method method abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain method method method method method method method method method method method method abstain abstain abstain method abstain abstain abstain method method abstain method abstain method abstain abstain method method method method abstain method method method method method method method method method method abstain abstain method method abstain abstain method method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain method method method method abstain method method method abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain result method objective abstain abstain abstain objective other other other other