Previous work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is on the quality of vision models. In this work, we investigate the impact of vision models on MMT. Given the fact that Transformer is becoming popular in computer vision, we experiment with various strong models (such as Vision Transformer) and enhanced features (such as object-detection and image captioning). We develop a selective attention model to study the patch-level contribution of an image in MMT. On detailed probing tasks, we find that stronger vision models are helpful for learning translation from the visual modality. Our results also suggest the need of carefully examining MMT models, especially when current benchmarks are small-scale and biased. Our code could be found at https: //github.com/libeineu/fairseq_mmt . Multimodal machine translation (MMT) has emerged as an active field of research which marries the worlds of computer vision (CV) and natural language processing (NLP) (Specia et al., 2016). Early models of this kind produce a translation given the fused representation of both the visual and textual inputs (Caglayan et al., 2016; Libovick and Helcl, 2017; Calixto and Liu, 2017). As expected, such a paradigm achieves promising BLEU improvements and inspires the community to follow up. But soon researchers found that MMT systems did not act as what they ordinarily designed: the visual modality contributes to translation little. For example, it is not harmful to MMT systems when the input image is irrelevant to the text (Grnroos et al., 2018; Lala et al., 2018), or even when the vision features are absent (Elliott, 2018). More recently, Wu et al. (2021) have pointed out that the  Corresponding author. use of the visual modality is a way of regularization for training but not a complement to the text modality. As another response to the analysis of MMT, Caglayan et al. (2019) investigate how the vision features correlate to the text. They find that the input image helps translation when some of the input words are masked. Note that previous work has for the most part focused on integrating off-the-shelf vision models (such as ResNet-50) into MMT. The underlying assumption here is that the existing vision models are powerful enough to encode the image. This implicitly ignores the quality of vision models in representing images. But computer vision is facing a new trend by moving from CNNs to Transformer as the backbone model (Dosovitskiy et al., 2021; Liu et al., 2021b; Carion et al., 2020). A natural question that arises is: how will MMT systems behave if stronger vision models are adopted? In this work, we address this question by a systematic study of using various vision models in MMT, in particular using the most successful models in recent studies (such as Vision Transformer, or ViT for short). We find that the patch method used in Transformer-based vision models offers an opportunity to detail the patch-level contribution of the image. This leads us to develop a selective attention model to correlate words with image patches. Beyond this, we introduce object-detection and image captioning features into MMT for further improvements of the vision models (Carion et al., 2020; Fang et al., 2021). Following Caglayan et al. (2019)'s work, we design more detailed probing tasks to examine to what degree the visual modality contributes to MMT. We run an extensive set of experiments on En-De and En-Fr MMT tasks. Our findings are  Stronger vision models help. For example, ViT can beat ResNet-50 on the probing tasks though the superiority is not significant on standard MMT data. Automatic evaluation on current MMT tasks might not be a good indicator for the effectiveness of MMT models. For example, models enhanced with object-detection and image captioning features yield good BLEU scores on the original MMT task but show modest or no contributions on the probing tasks. We start with a description of the probing tasks. It is followed by a design of vision features and a selective attention mechanism for introducing ViT-like representations into MMT. To know how an image contributes to translation, a way is to mask some of the input words (call this insufficient text) and force the translation model to learn from the image. Following the previous design of color deprivation and entity-based masking, we present detailed probing tasks which are complementary to Caglayan et al. (2019)'s work. In preliminary experiments 1 , we find that color, character and noun are three kinds of words which could be complemented according to the visual modality once the corresponding texts are masked. The following probing tasks are designed accordingly. Color-based Probing In training, all source words referring to a color are replaced by a special token [ Mask_C ] . There are 8 , 919 sentences involving color words, and nearly one third of them involve more than one color. It is worth noting that each color may have two or more translations due to the rich morphology in German and French. For example, the English green can be translated to 1 We choose the Multi30K En-De and En-Fr datasets for experiments. grn, grne, grnes, grner, grnen and grnem in German. We design two criteria to measure the accuracy of translation. The first criterion is strict. The correct translation requires generating the same color and the same gender as in reference translations. The second criterion is relaxed and all translations expressing the same color are correct. Character-based Probing For character words, we choose man, woman, people, men, girl and boy. More than 60% sentences contain character words in our training data, so they are reasonable indicators of assessing the ability to infer correct translations from the input image. Here we use [ MASK_P ] for masking. Note that some character words have more than two translations, e.g. people, we also use the same evaluation metric with the color-based probing task, including relaxed and strict two criteria. Noun-based Probing For more complex scenarios, a sentence can be masked with several kinds of ambiguous words, such as animals, clothing, and vehicles, provided by Flickr30K (Plummer et al., 2015). High-frequency words labeled with noun (or nouns) are more likely to be masked as [ MASK_N ] (or [ MASK_NS ])). See Table 1 for example insufficient text with different numbers of masks. In addition to ResNet-50, we choose Transformer-based vision models. General Backbone. Vision Transformer (ViT) and Swin Transformer are popular models in computer vision (Dosovitskiy et al., 2021; Liu et al., 2021b). We use ViT with various model capacities to vary from weak to strong ViT models. Object-detection. For pretrained object-detection vision models, we choose DETR (Carion et al., 2020) and QueryInst (Fang et al., 2021) for their strong performance. Image Captioning. For image captioning models, we choose CATR 2 because it is a Transformer-based image captioning architecture and can be easily implemented on top of ViT. We form a number of vision features by combining the methods described above. More details are presented in Section 3. ViT and related models perform in almost the same way as Transformer in NLP (Vaswani et al., 2017). Unlike the general models in CV, ViT does not represent the image as a single vector. Instead, it generates a sequence of patches for image representation. An advantage of this design is that we can use the attention mechanism to correlate image patches to words. Thus, we present a selective attention model to model the patch-level contribution of the image. See Figure 1 for the architecture. Text-only Transformer Transformer follows an encoder-decoder paradigm (the purple region in Figure 1) . The encoder is a stack of identical layers. Each layer consists of a self-attention (SAN) block and a feedforward network (FFN) block. The decoder shares a similar design with the encoder, but with an additional cross-attention block. Gated Fusion Gated fusion mechanism is a popular technique for fusing representations from different sources (Wu et al., 2021; Zhang et al., 2020; Lin et al., 2020; Yin et al., 2020). Given the text 2 https://github.com/saahiluppal/catr input X text and the image input X img , the text representation H text and the image feature H img can be defined as: H text = TransformerEncoder ( X text ) (1) H img = W ViT ( X img ) (2) where W is a projection matrix to convert the shape of ViT ( X img ) into that of H text . Note that ViT (  ) can be replaced by other vision models, e.g. DETR, Swin Transformer and etc. Then, the gate   [0 , 1] and the fuzed output are defined as:  = Sigmoid ( UH text + V H img ) (3) H Out = (1   )  H text +   H img (4) where U and V are trainable variables. controls how much visual information is kept. Then, the fusion vector H Out is fed into the decoder. See the right side of the pink region in Figure 1 for an illustration of the gated fusion models. Selective Attention After obtaining the text and image representations (or features), we use a single-head attention network to correlate words with image patches, where the query, key and value are H text , H img and H img , respectively. Then the selective attention output H img attn is defined to be: H img attn = Softmax ( QKT  d k ) V (5) where d k is the same as the dimension of H text because a single head is used. Then the fused representation could be obtained by using Eqs. 3 and 4 and replacing H img with H img attn . We conducted experiments on the widely used Multi30K benchmark (Elliott et al., 2016). The training and validation sets consisted of 29 , 000 and 1 , 014 instances, respectively. We reported the results on the Test2016, Test2017 and MSCOCO test sets (Elliott et al., 2017). Note that MSCOCO is more challenging for MMT models due to the out-of-domain instances with ambiguous verbs. Following the setup in (Wu et al., 2021), we learned a joint BPE code for 10 , 000 merging operations for both the source and target languages, resulting in vocabularies of 9 , 716 and 9 , 548 entries for the En-De and En-Fr tasks. We followed the Wu et al. (2021)'s work to conduct experiments with Transformer-Tiny configu-ration, which is more suited for small datasets like Multi30K. Note that smaller models even obtain higher BLEU scores than pervious MMT models. Similar observations have been discussed when building context-aware machine translation models (Li et al., 2020). The model consists of 4 encoder and decoder layers. The hidden size is 128 and the filter size of FFN is 256 . There are 4 heads in the multi-head self-attention mechanism. We set the dropout as 0 . 3 and the label smoothing as 0 . 1 . Our implementation was based on Fairseq (Ott et al., 2019). For training, we used Adam Optimizer (Kingma and Ba, 2015) with  1 = 0 . 9 ,  2 = 0 . 98 and (cid:15) = 10  8 . We adopted the same learning rate schedule as (Vaswani et al., 2017), where the learning rate first increased linearly for warmup = 2000 steps from 1 e  7 to 5 e  3 . After the warmup, the learning rate decayed proportionally to the inverse square root of the current step. Each training batch contained 4 , 096 tokens. We also adopted the early-stop training strategy (Zhang et al., 2020) to avoid the overfitting issue. For evaluation, we averaged the last 10 checkpoints for more reliable results. The width of beam size was set to 5 . The performance was measured by BLEU and METEOR for all test sets. Also, we used accuracy for evaluation on the probing tasks. Table 2 summarizes the results on standard MMT data. Each model was evaluated on three test sets on two language pairs. We see, first of all, that the improvements of previous methods (Rows 2-4) over the tiny baseline are marginal in terms of both BLEU and METEOR. This confirms the assumption that the visual features are not fully used if the text is complete (Caglayan et al., 2019). When switching the vision features from ResNet (Row.5) to ViT (Row.6), there are no significant BLEU gains. Then, we test them on the proposed probing tasks to examine the real contribution to MMT. Color-based Probing Table 3 shows the accuracy on the color-based probing task. We see that the accuracy improvement of the gated fusion method is marginal by both restrict and relaxed criteria. However, replacing ResNet with ViT yields gains of over 8 accuracy points across three test 6330 Systems Test2016 Test2017 MSCOCO Restrict Relaxed Restrict Relaxed Restrict Relaxed English  German Text-only Transformer 25.93 34.42 22.57 35.70 18.75 23.44 Gated Fusion + ResNet 27.23 (  1.30) 35.51 (  1.09) 23.10 (  0.53) 37.01 (  1.31) 21.88 (  3.13) 25.00 (  1.56) Gated Fusion + ViT 35.08 (  9.15) 42.48 (  8.06) 25.46 (  2.89) 41.73 (  6.03) 25.00 (  6.25) 31.25 (  7.81) Selective Attn + ViT 51.20 (  25.27 ) 64.71 (  30.29 ) 31.76 (  9.19 ) 53.54 (  17.84 ) 43.75 (  25.00 ) 56.25 (  32.81 ) English  French Text-only Transformer 30.72 33.12 34.91 38.85 23.44 29.69 Gated Fusion + ResNet 32.68 (  1.96) 35.51 (  2.39) 32.55 (  2.36) 35.17 (  3.68) 17.19 (  6.25) 23.44 (  6.25) Gated Fusion + ViT 45.53 (  14.81) 50.76 (  17.64) 45.41 (  10.50) 52.23 (  13.38) 34.38 (  10.94) 43.75 (  14.06) Selective Attn + ViT 62.96 (  32.24 ) 68.85 (  35.73 ) 49.34 (  14.43 ) 55.38 (  16.53 ) 43.75 (  20.31 ) 53.12 (  23.43 ) Table 3: The accuracy of MMT systems when applied color-based probing. sets on En-De task. Similar improvements are observed on the En-Fr task. The finding here indicates that stronger vision features are helpful for representing the visual information. Moreover, selective attention can make better use of the ViT features, achieving over 20 accuracy gains on three test sets. This verifies the conjecture that the selective attention can further enhance the fused representation for the ViT features. Character-based Probing Table 4 shows similar results as in Table 3. ViT with selective attention performs the best on most scenarios, it is only slightly inferior to Gated Fusion + ViT on the MSCOCO dataset. While the gated fusion method with ResNet feature behaves far from desirable. It even underperforms the text-only Transformer, though the text-only Transformer is carefully regularized. A potential explanation is the character-based probing task is more challenging than the color-based probing task because it is more diffi-cult for the model to find the correct corresponding region of the masked character word and provide useful signals to the text encoder. the results on the En-De and En-Fr tasks, respectively. The ViT features can significantly outperform the ResNet features across all masking methods on the two language pairs. We also observe that the gap between the ResNet and ViT features is gradually enlarged as more nouns are masked. This confirms the results in (Dosovitskiy et al., 2021). We further explore the impact of model capacity. Here, we report the results of ViT and Swin Transformer because they are strong models in recent studies. Our conjecture here is that larger ViT/Swin models can describe the image more accurately, which enables the text encoder to receive richer complementary information. Figure 3 depicts the BLEU scores in progressive noun masking scenarios. Intuitively, larger ViT and Swin models provide more complementary knowledge to complete the insufficient text representations. Nevertheless, a counterintuitive phenomenon is the inferiority of Swin across all scenarios in the same configuration, though it outperforms ViT on most computer vision benchmarks. We attribute the reason to the short length of the patch sequence. In patch, ViT has a length of 577 (576 sequence 6331 Gated Fusion+ResNet: Gated Fusion+ViT_Large: Selective Attn+ViT_Large: segments and a special token CLS ) when the image resolution and the patch size are 384  384 and 16  16 . However, Swin has a fixed sequence length (49) restricted by the shifted window operation. This leads to more fine-grained local features for ViT, which is beneficial to the selective attention mechanism for extracting more relevant pieces. Then, we investigate the impact of the enhanced vision features on MMT. Previous studies have already attempted to leverage object-detection features (Zhao et al., 2020; Wang and Xiong, 2021) but the observation here is slightly different. Beyond the object-detection pretrained features, we also take the image captioning task into account. Rows 11-13 in Table 2 summarize the results of the three enhanced vision features on the standard MMT data, and Figure 4 depicts the results on insufficient texts. Here we choose ViT-Tiny-based models for comparison due to the similar model capacity they own 3 . We see that not only the object-detection (DETR and QueryInst), but also the image captioning (CATR) pretrained fea-3 Only pretrained vision models in a 256 hidden-size are available 6332 System Patch Reso. tures obtain superior performance compared with ViT-tiny (Row 8) when the text is complete. It is consistent with previous findings (Yin et al., 2020; Zhao et al., 2020). However, the advantages do not persist when switching to limited text scenarios. A possible explanation is that these methods are sensitive to the quality of the extracted objects. We leave this as future work. It is well-known that higher resolutions are beneficial to the accuracy improvement in computer vision tasks (Dosovitskiy et al., 2021). Despite the success of the Transformer architecture, recent studies show that the success of ViT mainly comes from the successful use of the patch schema (Dosovitskiy et al., 2021). Here, we compare MMT systems with different resolutions and patch sizes based on ViT-Base. The results on three probing tasks (see Table 5) again confirm the above assumption that fine-grained vision features are more suited for the selective attention. Also, the attention map visualized in Figure 5 demonstrates that high resolution with fine-grained patch schema can attend to correct regions of the image for each masked token. For example, both models pay the right attention to the masked character and noun, but the model with low resolution fails to detect the right region of color. The finding here may shed light to other multimodal tasks, such as VQA. Incongruent decoding is a widely used manner to evaluate whether the visual modality contributes to the text (Caglayan et al., 2019, 2021). Table 6 shows that incongruent decoding causes obvious BLEU drops except for the ResNet feature. ViT beats the ResNet with gated fusion. It yields higher BLEU scores with congruent decoding and exhibits a larger BLEU drop with incongruent decoding. We also find that the ViT features learned from scratch are also insensitive to the visual modality. This is reasonable that the learned vision systems are not sufficiently strong due to the data scarcity of Multi30K. Thus the visual modality acts more like noise signals. In addition, focusing on the results of pretrained selective attention + ViT, the gap between congruent and incongruent decoding gradually becomes larger. We also investigate whether the ensemble vision features can help. Concretely, we choose ViT and CATR to independently generate the fused representations with the text feature, and then the ensemble feature is obtained based on them. We see that the ensemble vision feature performs the best on the congruent decoding, and achieves the largest 6333 System Mask 1 Mask 2 Mask 3 Mask 4 Cong. BLEU gaps on four masking scenarios compared with other systems. These results again indicate that stronger visual contexts indeed help. Finally, we compare several real cases. We choose gated fusion ( CNN ) (Wu et al., 2021) and selective attention + ViT_Base ( ViT ) for comparison. The qualitative examples in Table 7 demonstrate that the visual modality is complementary rather than redundant if the text is insufficient. To figure out whether the German translation is right or not, we provide the human-translation results. First, we see the top half case of Table 7, ViT can fill in the masked entities and generate the correct translations even four entities were masked. Unfortunately, CNN incorrectly judges the man as a woman. Also, it cannot distinguish the right color of shirt due to the complex background. When given a more complex image (the bottom half case), it is still a challenge for ViT to generate the right translation. The observation here inspires us to design a more powerful fusion method. Also, the data scarcity problem is a root issue to prevent us from further improving the cross-modal translation quality. Multimodal machine translation is a cross-domain task in the field of machine translation. Early attempts mainly focused on enhancing the MMT model by better incorporation of the vision features (Calixto and Liu, 2017; Elliott and Kdr, 2017; Delbrouck and Dupont, 2017). However, directly encoding the whole image feature brings additional noise to the text (Yao and Wan, 2020; Liu et al., 2021a). To address the above issue, Yao and Wan (2020) proposed a multimodal self-attention to consider the relative difference of information between two modalities. Similarly, Liu et al. (2021a) used a Gumbel Softmax to achieve the same goal. Researchers also realize that the visual modality may be redundant. Irrelevant images have little impact on the translation quality, and no significant BLEU drop is observed even the image is absent (Elliott, 2018). Encouraging results appeared in 6334 Caglayan et al. (2019)'s work. They pointed out that the visual modality is still useful when the linguistic context is scarce, but is less sensitive when exposed to complete sentences. More recently, Wu et al. (2021) attributed the BLEU gain on MMT tasks to the regularization training, and they again emphasized the imperative of constructing proper insufficient textual input. It is worthy to note that the proposed probing task is an improved version based upon previous work (Caglayan et al., 2019; Wu et al., 2021). We also opensource the preprocessed data and the corresponding scripts for the subsequent researchers to experiment on. Another line of research is to explore large-scale cross-modal pretraining models. In this way, the MMT task is regarded as a downstream task. For example, CLIP (Radford et al., 2021) is a general cross-modal pretraining model, which learns to perform a wide variety of tasks via natural language prompting. Caglayan et al. (2021) presented a MMT-specific pretraining model which combines the translation language modeling with masked region classification objectives. In this work, we make a systematic study on whether stronger vision features are helpful. We also extend the research to enhanced features, such as object-detection and image captioning, which are complementary to previous work. In this work, we show that stronger vision features (e.g. ViT-like models) strengthen MMT systems on three proposed probing tasks. We present a selective attention method for ViT-based models to make better use of the patch-level representation. The result here shows a promising line of research on developing better vision models for multimodal tasks. As far as we know, this is the first attempt to build MMT systems with Transformer only. In future work, we are willing to investigate whether it is possible to use a single set of parameters to encode the vision and text modalities. This work was supported in part by the National Science Foundation of China (Nos. 61732005 and 61876035), the National Key R&D Project of China (No. 2019QY1801), the China HTRD Center Project (No. 2020AAA0107904) and Yunnan Provincial Major Science and Technology Special Plan Projects (Nos. 201902D08001905 and 202103AA080015). The authors would like to thank anonymous reviewers for their valuable comments. And thank Yufan Jiang for his helpful advice to improve the paper.
abstain objective method objective result result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective objective abstain abstain abstain abstain objective objective objective abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain method method abstain abstain abstain method abstain abstain method abstain method abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result method abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain other other other other other other other other other other other method other other other other abstain objective objective method abstain objective objective other other other other