Abstract Multimodal machine translation and textual chat translation have received considerable attention in recent years. Although the conversation in its natural form is usually multimodal, there still lacks work on multimodal machine translation in conversations. In this work, we introduce a new task named M ultimodal C hat T ranslation (MCT), aiming to generate more accurate translations with the help of the associated dialogue history and visual context. To this end, we firstly construct a M ultimodal S entiment C hat T ranslation D ataset (MSCTD) containing 142,871 English-Chinese utterance pairs in 14,762 bilingual dialogues and 30,370 English-German utterance pairs in 3,079 bilingual dialogues. Each utterance pair, corresponding to the visual context that reflects the current conversational scene, is annotated with a sentiment label. Then, we benchmark the task by establishing multiple baseline systems that incorporate multimodal and sentiment features for MCT. Preliminary experiments on four language directions (English  Chinese and English  German) verify the potential of contextual and multimodal information fusion and the positive impact of sentiment on the MCT task. Additionally, as a by-product of the MSCTD, it also provides two new benchmarks on multimodal dialogue sentiment analysis. Our work can facilitate research on both multimodal chat translation and multimodal dialogue sentiment analysis. 1 1 Introduction Multimodal machine translation (Huang et al., 2016; Calixto and Liu, 2017) and textual chat translation (Wang et al., 2016; Farajian et al., 2020; Liang et al., 2021a) mainly focus on investigating  Equal contribution. the potential visual features and dialogue context, respectively. Both of them have received much attention. Although plenty of studies on them have been carried out based on either image captions (Calixto et al., 2017, 2019; Ive et al., 2019; Yin et al., 2020; Yao and Wan, 2020) or textual dialogues (Wang et al., 2017; Maruf et al., 2018; Liang et al., 2021c), to our knowledge, little research work has been devoted to multimodal machine translation in conversations. One important reason is the lack of multimodal bilingual conversational datasets. Generally, conversation in its natural form is multimodal (Poria et al., 2019; Liang et al., 2021b). When humans converse, what a speaker would say next depends largely on what he/she sees. That is, the visual information plays a key role in ( i ) supplementing some crucial scene information ( e.g. , the specific locations or objects, or facial expressions), ( ii ) resolving ambiguous multi-sense words ( e.g. , bank), and ( iii ) addressing pronominal anaphora issues ( e.g. , it/this). For instance, as shown in Fig. 1 (a), the image obviously points out the current location on the sea, which may help disambiguate the meaning of course in the utterance X 5 . Specifically, the dialogue history ( i.e. , talking about mar-itime affairs) and the corresponding visual context ( i.e. , on the sea/boat) assist us to determine that the word course means route/direction instead of curriculum. In Fig. 1 (b), the visual context indicates object information, i.e. , the defibrillator in X 1 , which may help with translation. In Fig. 1 (c), the image of the utterance X 1 also demonstrates that it can provide appropriate candidates ( i.e., the jeans) when translating the pronoun these. Besides, the image offers some clues to judge the sentiment when it is hard to judge the polarity based only on the utterance ( e.g. , Y 2 in Fig. 1 (b) and X 3 in Fig. 1 (c)). All of the above call for a real-life multimodal bilingual conversational data resource that can encourage further research in chat transla-2601 Figure 1: Three examples of the annotated multimodal bilingual dialogue in our MSCTD and the conversation is going from left to right. In this work, we propose a new task named M ultimodal C hat T ranslation (MCT), with the goal to produce more accurate translations by taking the dialogue history and visual context into consideration. To this end, we firstly construct a M ultimodal S entiment C hat T ranslation D ataset (MSCTD). The MSCTD includes over 17k multimodal bilingual conversations (more than 142k English-Chinese and 30k English-German utterance pairs), where each utterance pair corresponds with the associated visual context indicating where it happens. In addition, each utterance is annotated with one sentiment label ( i.e. , posi-tive/neutral/negative). Based on the constructed MSCTD, we benchmark the MCT task by establishing multiple Transformer-based (Vaswani et al., 2017) systems adapted from several advanced representative multimodal machine translation models (Ive et al., 2019; Yao and Wan, 2020) and textual chat translation models (Ma et al., 2020; Liang et al., 2021c). Specifically, we incorporate multimodal features and sentiment features into these models for a suitable translation under the current conversational scene. Extensive experiments on four language directions (English  Chinese and English  German) in terms of BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014) and TER (Snover et al., 2006), demonstrate the effectiveness of contextual and multimodal information fusion, and the positive impact of sentiment on MCT. Furthermore, experiments on the multimodal dialogue sentiment analysis task of the three languages show the added value of the proposed MSCTD. In summary, our main contributions are:  We propose a new task: multimodal chat translation named MCT, to advance multimodal chat translation research. We are the first that contributes the human-annotated multimodal sentiment chat translation dataset (MSCTD), which contains 17,841 multimodal bilingual conversations, totally 173,240 <English utterance, Chinese/German utterance, image, sentiment> quadruplets. We implement multiple Transformer-based baselines and provide benchmarks for the new 2602 task. We also conduct comprehensive analysis and ablation study to offer more insights. As a by-product of our MSCTD, it also facilitates the development of multimodal dialogue sentiment analysis. In this section, we firstly clarify the symbol definition, and then define the proposed Multimodal Chat Translation task and the existing Multimodal Dialogue Sentiment Analysis task. In a multimodal bilingual conversation ( e.g. , Fig. 1 (a)), we assume the two speakers have alternatively given utterances in different languages for u turns, resulting in X 1 , X 2 , X 3 , X 4 , ..., X u and Y 1 , Y 2 , Y 3 , Y 4 , ..., Y u on the source and target sides, respectively, along with the corresponding visual context representing where it happens: Z 1 , Z 2 , Z 3 , Z 4 , ..., Z u . Among these utterances, X 1 , X 3 , X 5 , ..., X u are originally spoken by the first speaker and Y 1 , Y 3 , Y 5 , ..., Y u are the corresponding translations in the target language. Similarly, Y 2 , Y 4 , Y 6 , ..., Y u  1 are originally spoken by the second speaker and X 2 , X 4 , X 6 , ..., X u  1 are the translated utterances in the source language. According to languages and modalities, we define three types of context: (1) the dialogue history context of X u on the source side as CX u = { X 1 , X 2 , X 3 , ..., X u  1 } , and (2) that of Y u on the target side as CY u = { Y 1 , Y 2 , Y 3 , ..., Y u  1 } , and (3) the visual dialogue context CZ u = { Z 1 , Z 2 , Z 3 , ..., Z u  1 , Z u } . 2 Multimodal Chat Translation. When translating the u -th utterance X u = { x u, 1 , x u, 2 , ..., x u,N } , the goal of the MCT task is to generate Y u = { y u, 1 , y u, 2 , ..., y u,T } with the guidance of bilingual dialogue history contexts CX u and CY u and the associated visual context CZ u . Formally, the probability distribution of the target utterance Y u is defined as follows: P ( Y u | X u , C u ) = T (cid:89) t =1 p ( y u,t | y u,<t , X u , C u ) , (1) where y u,<t = { y u, 1 , y u, 2 , y u, 3 , ..., y u,t  1 } and C u = {C X u , CY u , CZ u } . Multimodal Dialogue Sentiment Analysis. Taking the u -th utterance X u for example, the task aims to predict a sentiment label   { Positive , Neutral , Negative } for it given the corresponding image Z u and the dialogue history CX u . In this section, we mainly introduce our MSCTD in five aspects: Data Source  3.1, Annotation Procedure  3.2, Annotation Quality Assessment  3.3, Dataset Statistics  3.4, and the introduction of Related Datasets  3.5. We mainly select the multimodal dialogues from the public available OpenViDial dataset (Meng et al., 2021), where each monolingual (English) utterance corresponds to an image. Since the original English utterance in OpenViDial is automatically extracted from the corresponding movie image by optical character recognition (OCR) 3 , it contains a lot of noises or errors. Furthermore, the lack of associated translations and sentiment labels for utterances, makes it impossible for directly conducting research on multimodal chat translation, sentiment-aware machine translation, and multimodal dialogue sentiment analysis with this data. Therefore, we further correct the wrong English utterances and annotate the corresponding Chinese/German translations and sentiment labels. includes two steps: automatic annotation and then human annotation according to the annotation rules. Automatic Annotation. To improve the annotation efficiency, we firstly construct a paired English-Chinese subtitle database 4 . Then, we utilize the original English utterance to automatically select its Chinese translation by perfectly matching the English subtitle in the constructed bilingual database. As a result, about 78.57% original English utterances are paired with Chinese translations. 4 To build this database, we firstly crawl two consecutive English and Chinese movie subtitles (not aligned) from here https://www.kexiaoguo.com/ . Then, we use several advanced technologies ( e.g. , Vecalign (Thompson and Koehn, 2019) and LASER (Schwenk, 2018)) to align these subtitles. Finally, we obtain the large-scale bilingual dialogue dataset (28M). We will also release this dataset, together with the MSCTD, to facilitate subsequent research. Human Annotation. Since the full data are large, we divide the data into three parts and employ three annotators who are Chinese postgraduate students highly proficient in English comprehension. Each annotator is responsible for annotating one part according to the following guidelines:  Check and correct each English utterance;  Check and correct the matched Chinese subtitle to suit the current conversational scene;  For the remaining 21.43% (without Chinese subtitles), translate them according to the corrected English utterance, the corresponding image, and the dialogue history. Additionally, we employ another three annotators to label sentiment polarity for each utterance independently ( i.e. , each one annotates the full data) according to the current utterance, the associated image and the dialogue history. Following Firdaus et al. (2020), majority voting scheme is used for selecting the final sentiment label for each utterance. Finally, having the conversations in both languages allows us to simulate bilingual conversations where one speaker speaks in English and the other responds in Chinese (Farajian et al., 2020; Liang et al., 2021a). Fig. 1 shows three bilingual conversations where the two speakers have alternatively given utterances, along with their corresponding translations. By doing so, we build the MSCTD 5 . 5 For English  German, we firstly sample a small set of training data and apply the same test and validation set with the English  Chinese version. Then, the German translations are collected from professional English-German workers con-3.3 Annotation Quality Assessment To evaluate the quality of annotation, we use Fleiss' Kappa to measure the overall annotation consistency among three annotators (Fleiss and Cohen, 1973). We measure this data from two aspects: translation quality and sentiment quality. For translation quality, we measure the inter-annotator agreement on a subset of data (sample 50 dialogues with 504 utterances), and we ask the three annotators mentioned above to re-annotate this subset independently. Then, we invite another postgraduate student to measure the inter-annotator agreement on the re-annotated subset by the three annotators. Finally, the inter-annotator agreement calculated by Fleiss' kappa are 0.921 for English  Chinese and 0.957 for English  German, respectively. They indicate Almost Perfect Agreement between three annotators. For sentiment quality, we measure the inter-annotator agreement on the full dataset. The inter-annotator agreements calculated by Fleiss' kappa is 0.695, which indicates Substantial Agreement between three annotators. The level is consistent with previous work (Firdaus et al., 2020) which can be considered as reliable. As shown in Tab. 1, the MSCTD contains totally 17,841 bilingual conversations and 142,871/30,370 tracted via a language service company (magicdatatech). The three crwodworkers are asked to translate them according to the English utterance, the corresponding image, and the dialogue history. English-Chinese/English-German utterance pairs with two modalities ( i.e. , text and image), where each utterance has been annotated with onesenti-ment label. For English-Chinese/English-German, we split the dialogues into 13,749/2,066 for train, 504/504 for valid, and 509/509 for test while keeping roughly the same distribution of the utterance pair/image, respectively. The detailed annotation of sentiment labels are also listed in Tab. 1, where three labels account for similar proportion. Based on the statistics in Tab. 1, the average number of turns per dialogue is about 10, and the average numbers of tokens per turn are 8.2, 10.9, and 8.3 for English utterances (word level), Chinese utterances (character level), and German utterance (word level), respectively. The related datasets mainly involve three research fields: multimodal machine translation, textual chat translation, and multimodal dialogue sentiment analysis. In multimodal machine translation , there exists one dataset: Multi30K (Elliott et al., 2016), where each image is paired with one English caption and two human translations into German and French. It is an extension of the original English description dataset: Flickr30K (Young et al., 2014). Afterwards, some small-scale multimodal test sets (about 3k instances) are released to evaluate the system, such as WMT18 test set (1,071 instances) (Barrault et al., 2018). In textual chat translation , three datasets have been released: BSD-AMI-ON (Rikters et al., 2020), BconTrasT (Farajian et al., 2020), and BMELD (Liang et al., 2021a). The BSD-AMI-ON is a document-aligned Japanese-English conversation corpus, which contains three sub-corpora: Business Scene Dialogue (BSD (Rikters et al., 2019)), Japanese translation of AMI meeting corpus (AMI (McCowan et al., 2005)), and Japanese translation of OntoNotes 5.0 (ON (Marcus et al.)). The BconTrast and BMELD are two human-annotated datasets, which are extended from monolingual textual dialogue datasets Taskmaster-1 (Byrne et al., 2019) and MELD (Poria et al., 2019), respectively. In multimodal dialogue sentiment analysis , the MELD (Poria et al., 2019) and MEISD (Fir-daus et al., 2020) datasets are publicly available. The MELD dataset is constructed by extending the EmotionLines (Hsu et al., 2018) from the scripts of the popular sitcom Friends . It is similar to MEISD, which is also built from famous English TV shows under different genres ( e.g. , Friends , Grey's Anatomy , The Big Bang Theory ). The resources mentioned above are extensively used in corresponding fields of research and they even cover some sub-tasks in MSCTD. However, our MSCTD is different from them in terms of both complexity and quantity. Firstly, multimodal machine translation datasets and textual chat translation datasets are either in multimodal or textual dialogue, while ours includes both. It is obvious that conducting multimodal machine translation in conversations is more challenging due to the more complex scene. Furthermore, MSCTD covers four language directions and contains more than 17k human-annotated utterances-image triplets, which is more than the sum of the annotated ones in Multi30K, BSD-AMI-ON, BconTrasT, and BMELD. Tab. 2 provides information on the number of available modality, dialogues, and their constituent utterances for all the five datasets. What is more, our MSCTD is also annotated with sentiment labels while they are not. Secondly, compared with two existing multimodal dialogue sentiment analysis datasets, MSCTD's quantity of English version is nearly ten-times of the annotated utterances in MEISD or MELD. More importantly, our MSCTD provides an equivalent Chinese multimodal dialogue senti-2605 Dataset # Dialogues # Utterances Train Valid Test Train Valid Test MELD (Poria et al., 2019) 1,039 114 280 9,989 1,109 2,610 MEISD (Firdaus et al., 2020) 702 93 205 14,040 1,860 4,100 MSCTD-Zh (Chinese version) 13,749 504 509 132,741 5,063 5,067 MSCTD-En (English version) 13,749 504 509 132,741 5,063 5,067 MSCTD-De (German version) 2,066 504 509 20,240 5,063 5,067 Table 3: Comparisons of four multimodal dialogue sentiment analysis datasets: MELD, MEISD, and our MSCTD on two languages. ment analysis dataset and a relatively small German counterpart. Tab. 3 shows the comparison for all the five datasets, i.e. , MELD, MEISD, and our MSCTD on three languages. Following previous work (Wang et al., 2018; Ive et al., 2019; Meng et al., 2021), we focus on two types of image representation, namely the coarse-grained spatial visual feature maps and the fine-grained object-based visual features. Coarse-grained Spatial Visual (CSV) Features. We use the ResNet-50 model (He et al., 2016) pre-trained on ImageNet (Deng et al., 2009) to extract a high-dimensional feature vector f j  R d c for image Z j . These features contain output activations for various filters while preserving spatial information. We refer to models that use such features as CSV . Fine-grained Object-based Visual (FOV) Features. Since using coarse-grained image features may be insufficient to model fine-grained visual elements in images including the specific locations, objects, and facial expressions, we use a bag-of-objects representation where the objects are obtained using an off-shelf Faster R-CNNs (Ren et al., 2015) pre-trained on Visual Genome (Krishna et al., 2017). Specifically, for an input image Z j , we obtain a set of detected objects from Faster R-CNNs, i.e. , O j = { o j, 1 , o j, 2 , o j, 3 , ..., o j,m } , where m is the number of extracted objects and o j,   R d f . Each object is captured by a dense feature representation, which can be mapped back to a bounding box / region ( i.e. , Region-of-Interest (ROI)). We refer to models that use such features as FOV . Both types of features have been used in various vision and language tasks such as multimodal dialogue sentiment analysis (Firdaus et al., 2020), image captioning (Xu et al., 2015; Shi et al., 2021), and multimodal machine translation (Ive et al., 2019; Lin et al., 2020; Su et al., 2021). To provide convincing benchmarks for the MSCTD, we perform experiments with multiple Transformer-based (Vaswani et al., 2017) models for the multimodal chat translation task. Additionally, we provide several baselines for the multimodal dialogue sentiment analysis task. According to different visual features, we divide the baselines into three categories: text only ( T ), text plus coarse visual features ( T + CSV ), and text plus fine-grained visual features ( T + FOV ). T : Trans. (Vaswani et al., 2017): the standard transformer model, which is a sentence-level neural machine translation (NMT) model (Yan et al., 2020; Meng and Zhang, 2019; Zhang et al., 2019), i.e. , regardless of the dialogue history. TCT (Ma et al., 2020): A unified document-level NMT model based on Transformer by sharing the first encoder layer to incorporate the dialogue history, which is used as the Textual Chat Translation (TCT) model by (Liang et al., 2021c). CA-TCT (Liang et al., 2021c): A multi-task learning model that uses several auxiliary tasks to help model generate coherence-aware translations. T+CSV : Trans.+Emb (Vaswani et al., 2017): it concatenates the image feature to the word embedding and then trains the sentence-level NMT model. Trans.+Sum (Ive et al., 2019): it adds the projected image feature to each position of the encoder output. Trans.+Att (Ive et al., 2019): this model utilizes an additional cross-attention sublayer to attend the image features in each decoder block. MCT: we implement the multimodal self-attention (Yao and Wan, 2020) in the encoder to incorporate the image features into the chat translation model. CA-MCT: similarly, we incorporate image features into the multitask-based chat translation model (Liang et al., 2021c) by the multimodal self-attention. T+FOV : Trans.+Con (Vaswani et al., 2017): it concatenates the word sequence to the extracted object sequence and thus obtains a new sequence taken as the input of the sentence-level NMT model. Trans.+Obj (Ive et al., 2019): it is a translate-and-refine model (two-stage decoder) where the images are only used by a second-pass decoder. M-Trans. (Yao and Wan, 2020): it leverages a multimodal self-attention layer to encode multimodal information where the hidden representation of im-2606 Modality M# Model Chinese  English English  Chinese German  English English  German BLEU  METEOR  TER  BLEU  METEOR  TER  BLEU  METEOR  TER  BLEU  METEOR  TER  T M1 Trans. ages are induced from the text under the guidance of image-aware attention. MCT: here, we incorporate the object-level features into the model instead of coarse one. CA-MCT: similarly, we incorporate the object-level features into the multi-task learning model. We perform several experiments with different models. text-CNN (Kim, 2014): it only applies CNNs to extract textual information for each utterance in a dialogue. In this approach, we do not use the dialogue history or the additional visual information. DialogueRNN (Majumder et al., 2019): this baseline is a powerful approach for capturing dialogue history with effective mechanisms for sentiment analysis. DialogueRNN + BERT (Fir-daus et al., 2020): this model improves the performance of DialogueRNN by using BERT (Devlin et al., 2019) embeddings instead of Glove (Pen-nington et al., 2014) embeddings to represent the textual features. DialogueRNN + PLM: we propose a stronger baseline built upon the DialogueRNN for sentiment analysis. Specifically, we utilize RoBERTa (Liu et al., 2019) embeddings for English sentiment analysis, and ERNIE (Sun et al., 2019) embeddings for Chinese sentiment analysis, and XLM-R (Conneau et al., 2020) embeddings for German sentiment analysis. For multimodal chat translation, we utilize the standard Transformer-Base architecture (Vaswani et al., 2017). Generally, we use the settings described in previous work (Ive et al., 2019; Yao and Wan, 2020; Liang et al., 2021c) to conduct experiments on our MSCTD. For multimodal dialogue sentiment analysis, we mainly follow the settings of previous work (Poria et al., 2019; Firdaus et al., 2020). Please refer to Appendix A for more details. For multimodal chat translation, following previous work (Liang et al., 2021c; Ive et al., 2019), we use the SacreBLEU 6 (Post, 2018), METEOR (Denkowski and Lavie, 2014) and TER (Snover et al., 2006) with the statistical significance test (Koehn, 2004) for fair comparison. Specifically, for Chinese  English, we report case-insensitive score. For English  Chinese, the reported score is at the character level. For English  German, we report case-sensitive BLEU score. For multimodal dialogue sentiment analysis, following Poria et al. (2019), we report weighted-average F-score. Results on English  Chinese. (1) Among all only text-based models (M1  M3), we find that M1 6 BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+ version.1.4.13 2607 Model Chinese  English BLEU  METEOR  TER  Transformer ( T ) 20.43 24.06 61.00 TCT ( T ) 20.81 24.45 61.19 CA-TCT ( T ) 21.23 24.82 60.75 MCT ( T+CSV ) 22.25 25.60 59.69 CA-MCT ( T+CSV ) 22.68 25.60 59.14 Table 5: Sentiment-aware translation results using ground truth. performs worse than M2, showing that the dialogue history indeed is beneficial for better translations. Furthermore, M3 can further improve the translation performance, which suggests that modeling the coherence characteristic in conversations is crucial for higher results. These can also be found in other settings ( e.g. , M7 vs. M4  6; M8 vs. M7). (2) The models with image features incorporated get higher results than corresponding text-based models ( i.e. , M4  M6&M9 vs. M1; M7&M12 vs. M2; M8&M13 vs. M3). (3) The dialogue history and the image features obtain significant cumulative benefits (M8 vs. M1 and M13 vs. M1) (4) Among these image-based models (M4  M8 or M9  M13), we observe that different fusion manners of text and image features reflect great difference on effects. It shows that there is much room for further improvement using other more advanced fusion methods. (5) Using FOV image features is generally better than the coarse counterpart CSV (M9  M13 vs. M4  M8), which demonstrates that the fine-grained object elements may offer more specific and effective information for better translations. Results on English  German. Similar findings are found on English  German. This shows that our conclusions are solid and convincing on general datasets. All these results prove the value of our constructed MSCTD. Furthermore, we provide some stronger baselines that we firstly train the model on the general-domain corpus and then fine-tune it on our chat translation dataset. The results are presented in Table Tab. 8 of Appendix B, which show similar findings observed in Table Tab. 4. To evaluate the effect of sentiment, we conduct some experiments on several baselines including single-modality ones and double-modality ones. In Model Chinese  English BLEU  METEOR  TER  ACC. terms of implementation, following Si et al. (2019), we append the sentiment label to the head of the source utterance. Tab. 5 shows the results. Comparing them with the results (M1  M3 and M7  M8) without using the sentiment in Tab. 4, we find that using the ground-truth sentiment label has a positive impact on the translation performance. Therefore, we believe that it is a topic worthy of research in the future. We also conducted the experiments with automatically predicted sentiment labels rather than the gold ones as the reviewer suggested, where we used the mixed sentiment presentation by dot-multiplying the predicted sentiment distribution and the sentiment label representation. The results are shown in Tab. 6, where we find that the sentiment factor, as the inherent property of conversations, indeed has a positive impact on translation performance. We also observe that using the automatically predicted sentiment labels (actually the mixed sentiment representation) shows slightly lower results than using ground truth in terms of three metrics. The reason may be that the mixed sentiment representation has certain fault tolerance. Results on MSCTD-Zh. We can see that the text-based models perform much poorer than other multimodal systems, which shows that it is not enough to evaluate the sentiment based only on the text. It indicates that visual information and contextual embeddings are crucial for classifying sentiment polarities. Overall, we achieve weighted F1 score of 67.57% with the DialogueRNN+ERNIE model. Chinese. These show that it is beneficial to introduce the visual information and contextual embeddings into the multimodal dialogue sentiment analysis task for different languages. Overall, we achieve the best F1 score of 66.45% and 54.46% on English and German, respectively. On this task, we obtain consistent results with previous work (Poria et al., 2019; Firdaus et al., 2020), which suggests the utility and reliability of our MSCTD. Additionally, MSCTD-Zh and MSCTD-De bridge the gap on multimodal dialogue sentiment analysis of Chinese and German. In this paper, we introduce a new multimodal machine translation task in conversations. Then, we construct a multimodal sentiment chat translation dataset named MSCTD. Finally, we establish multiple baseline systems and demonstrate the importance of dialogue history and multimodal information for MCT task. Additionally, we conduct multimodal dialogue sentiment analysis task on three languages of the MSCTD to show its added value. MCT is a challenging task due to the complex scene in the MSCTD, leaving much room for further improvements. This work mainly focuses on introducing the new task and dataset, and we provide multiple models to benchmark the task. In the future, the following issues may be worth exploring to promote the performance of MCT:  How to effectively perceive and understand the visual scenes to better assist multimodal machine translation in conversations? How to build a multimodal conversation representation model to effectively align, interact, and fuse the information of two modalities? In this section, we discuss the main ethical considerations of MSCTD: (1) Intellectual property protection. The English utterance and image of MSCTD is from OpenViDial dataset (Meng et al., 2021). For our translation and sentiments, its permissions are granted to copy, distribute and modify the contents under the terms of the Creative Commons AttributionShareAlike 3.0 Unported License and Creative Commons CC0 License, respectively. (2) Privacy. The data source are publicly available movies. Its collection and Chinese/German annotation procedure is designed for chat translation purpose, and does not involve privacy issues. (3) Compensation. During the sentiment annotation, Chinese and German translation, the salary for annotating each utterance is determined by the average time of annotation and local labor compensation standard. (4) Data characteristics. We refer readers to the content and Meng et al. (2021) for more detailed characteristics. (5) Potential problems. While principled measures are taken to ensure the quality of the dataset, there might still be potential problems with the dataset quality, which may lead to incorrect translations in applications. However, moderate noise is common in large-scale modern translators, even for human translated sentences, which should not cause serious issues. This work is supported by the National Key R&D Program of China (2020AAA0108001) and the National Nature Science Foundation of China (No. 61976015, 61976016, 61876198 and 61370130). Liang is supported by 2021 Tencent Rhino-Bird Research Elite Training Program. The authors would like to thank the anonymous reviewers for their valuable comments to improve this paper.
abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other method abstain abstain abstain method abstain method abstain method abstain method method method method method method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain abstain result abstain result abstain abstain abstain abstain abstain objective method objective result abstain objective abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other