Neural chat translation aims to translate bilingual conversational text, which has a broad application in international exchanges and cooperation. Despite the impressive performance of sentence-level and context-aware Neural Machine Translation (NMT), there still remain challenges to translate bilingual conversational text due to its inherent characteristics such as role preference, dialogue coherence, and translation consistency. In this paper, we aim to promote the translation quality of conversational text by modeling the above properties. Specifically, we design three latent variational modules to learn the distributions of bilingual conversational characteristics. Through sampling from these learned distributions, the latent variables, tailored for role preference, dialogue coherence, and translation consistency, are incorporated into the NMT model for better translation. We evaluate our approach on the benchmark dataset BConTrasT (English  German) and a self-collected bilingual dialogue corpus, named BMELD (English  Chinese). Extensive experiments show that our approach notably boosts the performance over strong baselines by a large margin and significantly surpasses some state-of-the-art context-aware NMT models in terms of BLEU and TER. Additionally, we make the BMELD dataset publicly available for the research community. 1 1 Introduction A conversation may involve participants that speak in different languages ( e.g. , one speaking in English and another in Chinese). Fig. 1 shows an example, where the English role R 1 and the Chinese role R 2 are talking about the  boat . The  Work was done when Yunlong Liang was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. Jinan Xu is the corresponding author. 1 Code and data are publicly available at: https:// github.com/XL2248/CPCC Y 2 : n hu jish fnchun goal of chat translation is to translate bilingual conversational text, i.e. , converting one participant's language ( e.g. , English) to another's ( e.g. , Chinese) and vice versa (Farajian et al., 2020). It enables multiple speakers to communicate with each other in their native languages, which has a wide application in industry-level services. Although sentence-level Neural Machine Translation (NMT) (Sutskever et al., 2014; Vaswani et al., 2017; Meng and Zhang, 2019; Hassan et al., 2018; Yan et al., 2020; Zhang et al., 2019) has achieved promising progress, it still faces challenges in accurately translating conversational text due to abandoning the dialogue history, which leads to role-irrelevant, incoherent and inconsistent translations (Mirkin et al., 2015; Wang et al., 2017a; Laubli et al., 2018; Toral et al., 2018). Further, context-aware NMT (Tiedemann and Scherrer, 2017; Voita et al., 2018, 2019a,b; Wang et al., 2019; Maruf and Haffari, 2018; Maruf et al., 2019; Ma et al., 2020) can be directly applied to chat translation through incorporating the dialogue history but cannot obtain satisfactory results in this scenario (Moghe et al., 2020). One important reason is the lack of explicitly modeling the inherent bilingual conversational characteristics, e.g. , role preference, dialogue coherence, and translation consistency, as pointed out by Farajian et al. (2020). For a conversation, its dialogue history contains rich role preference information such as emotion, style, and humor, which is beneficial to role-relevant utterance generation (Wu et al., 2020). As shown in Fig. 1, the utterances X 1 , X 3 and X 5 from role R 1 always have strong emotions ( i.e. , joy ) because of his/her preference, and preserving the same preference information across languages can help raise emotional resonance and mutual understanding (Moghe et al., 2020). Meanwhile, there exists semantic coherence in the conversation, as the solid green arrow in Fig. 1, where the utterance X 5 naturally and semantically connects with the dialogue history ( X 1  4 ) on the topic  boat . In addition, the bilingual conversation exhibits translation consistency, where the correct lexical choice to translate the current utterance might have appeared in preceding turns. For instance, the word  sail  in X 1 is translated into  ji`achuan , and thus the word  sailing  in X 3 should be mapped into  ji`achuan  rather than other words ( e.g. ,  hangxng  2 ) to maintain translation consistency. On the contrary, if we ignore these characteristics, translations might be role-irrelevant, incoherent, inconsistent, and detrimental to further communication like the translation produced by the  S-NMT  in Fig. 1. Although the translation is acceptable at the sentence level, it is abrupt at the bilingual conversation level. Apparently, how to effectively exploit these bilingual conversational characteristics is one of the core issues in chat translation. And it is challenging to implicitly capture these properties by just incorporating the complex dialogue history into encoders due to lacking the relevant information guidance (Farajian et al., 2020). On the other hand, the Conditional Variational Auto-Encoder (CVAE) (Sohn et al., 2015) has shown its superiority in learning distributions of data properties, which is often utilized to model the diversity (Zhao et al., 2017), coherence (Wang and Wan, 2019) and users' personalities (Bak and Oh, 2019), etc. In spite of its success, adapting it to chat translation is non-trivial, especially involving multiple tailored latent variables. Therefore, in this paper, we propose a model, named CPCC, to capture role preference, dialogue coherence, and translation consistency with latent variables learned by the CVAE for neural chat translation. CPCC contains three specific latent variational modules to learn the distributions of role preference, dialogue coherence, and translation consistency, respectively. Specifically, we firstly use one role-tailored latent variable, sampled from the learned distribution conditioned only on the utterances from this role, to preserve preference. Then, we utilize another latent variable, generated by the distribution conditioned on source-language dialogue history, to maintain coherence. Finally, we leverage the last latent variable, generated by the distribution conditioned on paired bilingual conversational utterances, to keep translation consistency. As a result, these tailored latent variables allow our CPCC to produce role-specific, coherent, and consistent translations, and hence make the bilingual conversation go fluently. We conduct experiments on WMT20 Chat Translation dataset: BConTrasT (En  De 3 ) (Farajian et al., 2020) and a self-collected dialogue corpus: BMELD (En  Ch). Results demonstrate that our model achieves consistent improvements in four directions in terms of BLEU (Papineni et al., 2002) and TER (Snover et al., 2006), showing its effectiveness and generalizability. Human evaluation further suggests that our model effectively alleviates the issue of role-irrelevant, incoherent and inconsistent translations compared to other methods. Our contributions are summarized as follows:  To the best of our knowledge, we are the first to incorporate the role preference, dialogue coherence, and translation consistency into neural chat translation. We are the first to build a bridge between the dialogue and machine translation via conditional variational auto-encoder, which effectively models three inherent characteristics in bilingual conversation for neural chat translation. Our approach gains consistent and significant performance over the standard context-aware baseline and remarkably outperforms some state-of-the-art context-aware NMT models. We contribute a new bilingual dialogue corpus (BMELD, En  Ch) with manual translations and our codes to the research community. 3 English  German:En  De. English  Chinese:En  Ch. Given an input sentence X = { x i } Mi =1 with M tokens, the model is asked to produce its translation Y = { y i } Ni =1 with N tokens. The conditional distribution of the NMT is: p  ( Y | X ) = N (cid:89) t =1 p  ( y t | X, y 1: t  1 ) , where  are model parameters and y 1: t  1 is the partial translation. Given a source context DX = { X i } Ji =1 and a target context DY = { Y i } Ji =1 with J aligned sentence pairs ( X i , Y i ), the context-aware NMT (Ma et al., 2020) is formalized as: The variational NMT model (Zhang et al., 2016) is the combination of CVAE (Sohn et al., 2015) and NMT. It introduces a random latent variable z into the NMT conditional distribution: p  ( Y | X ) = (cid:90) z p  ( Y | X, z )  p  ( z | X ) d z . (1) Given a source sentence X , a latent variable z is firstly sampled by the prior network from the encoder, and then target sentence is generated by the decoder: Y  p  ( Y | X, z ) , where z  p  ( z | X ) . where  are parameters of the posterior network and KL(  ) indicates KullbackLeibler divergence between two distributions produced by prior networks and posterior networks (Sohn et al., 2015; Kingma and Welling, 2013). We aim to learn a model that can capture inherent characteristics in the bilingual dialogue history for producing high-quality translations, i.e. , using the context for better translations (Farajian et al., 2020). Following (Maruf et al., 2018), we define paired bilingual utterances ( X i , Y i ) as a turn in Fig. 2, where we will translate the current utterance X 2 k +1 at the (2 k + 1) -th turn. Here, we denote the utterance X 2 k +1 as X u and its translation Y 2 k +1 as Y u for simplicity, where X u = { x i } mi =1 with m tokens and Y u = { y i } ni =1 with n tokens. Formally, the conditional distribution for the current utterance is p  ( Y u | X u , C ) = n (cid:89) t =1 p  ( y t | X u , y 1: t  1 , C ) , where C is the bilingual dialogue history. Before we dig into the details of how to utilize C , we define three types of context in C (as shown in Fig. 2): (1) the set of previous role-specific source-language turns, denoted as C roleX = { X 1 , X 3 , X 5 , ..., X 2 k +1 } 4 where k  [0 , | T | 3 2 ] and T is the total number of turns; (2) the set of previous source-language turns, denoted as CX = { X 1 , X 2 , X 3 , ..., X 2 k } ; and (3) the set of previous target-language turns, denoted as CY = { Y 1 , Y 2 , Y 3 , ..., Y 2 k } . Fig. 3 demonstrates an overview of our model, consisting of five components: input representation , encoder , latent variational modules , decoder , and training objectives . Specifically, we aim to model both dialogue and translation simultaneously. Therefore, for the input representation ( 4.1), we incorporate dialogue-level embeddings, i.e. , role and dialogue turn embeddings, into the encoder ( 4.2). Then, we introduce three specific latent variational modules ( 4.3) to learn the distributions for varied inherent bilingual characteristics. Finally, we elaborate on how to incorporate the three tailored latent variables sampled from 4 C roleY = { Y 2 , Y 4 , Y 6 , ..., Y 2 k } is also role-specific utterances of the interlocutor, which is used to model the inter-locutor's consistency in the reverse translation direction. Here, we take one translation direction ( i.e. , En  Ch) as an example. the distributions into the decoder ( 4.4) and our two-stage training objectives ( 4.5). The CPCC contains three types of inputs: source input X u , target input Y u , and context inputs { C roleX , CX , CY } . Apart from the conventional word embeddings WE and position embeddings PE (Vaswani et al., 2017), we also introduce role embeddings RE and dialogue turn embeddings TE to identify different utterances. Specifically, for X u , we firstly project it into these embeddings. Then, we perform a sum operation to unify them into a single input for each token x i : h 0 i = WE ( x i ) + PE ( x i ) + RE ( x i ) + TE ( x i ) , (2) where 1  i  m and WE  R | V | d , RE  R | R | d and SE  R | T | d . | V | , | R | , | T | , and d denote the size of shared vocabulary, number of roles, max turns of dialogue, and hidden size, respectively. h 0  R m  d , similarly for Y u . For each of { C roleX , CX , CY } , we add [cls]' tag at the head of it and use [sep]' tag to separate its utterances (Devlin et al., 2019), and then get its embeddings via Eq. 2. 4.2 Encoder The Transformer encoder consists of N e stacked layers and each layer includes two sub-layers: 5 a multi-head self-attention ( SelfAtt ) sub-layer and a position-wise feed-forward network ( FFN ) sublayer (Vaswani et al., 2017): s (cid:96)e = SelfAtt( h (cid:96)  1 e ) + h (cid:96)  1 e , h (cid:96)  1 e  R m  d , h (cid:96)e = FFN( s (cid:96)e ) + s (cid:96)e , { h (cid:96)e , s (cid:96)e }  R m  d , 5 We omit the layer normalization for simplicity, and you may refer to (Vaswani et al., 2017) for more details. where h (cid:96)e denotes the state of the (cid:96) -th encoder layer and h 0 e denotes the initialized feature h 0 . We prepare the representations of X u and { C roleX , CX , CY } for training prior and recognition networks. For X u , we apply mean-pooling with mask operation over the output h N e ,X e of the N e th encoder layer, i.e. , h X = 1 m (cid:80) mi =1 ( M Xi h N e ,X e,i ) , h X  R d , where MX  R m denotes the mask matrix, whose value is either 1 or 0 indicating whether the token is padded (Zhang et al., 2016). For C roleX , as shown in Fig. 3, we follow (Ma et al., 2020) and share the first encoder layer to obtain the context representation. Here, we take the hidden state of [cls]' as its representation, denoted as h ctxrole  R d . Similarly, we obtain representations of CX and CY , denoted as h ctxX  R d and h ctxY  R d , respectively. For training recognition networks, we obtain the representation of Y u as h Y = 1 n (cid:80) ni =1 ( M Yi h N e ,Y e,i ) , h Y  R d , where MY  R n , similar to MX . We design three tailored latent variational modules to learn the distributions of inherent bilingual conversational characteristics, i.e. , role preference, dialogue coherence, and translation consistency. Role Preference. To preserve the role preference when translating the role's current utterance, we only encode the previous utterances of this role and produce a role-tailored latent variable z role  R d z , where d z is the latent size. Inspired by (Wang and Wan, 2019), we use isotropic Gaussian distribution as the prior distribution of z role : p  ( z role | X u , C roleX )  N (  role ,  2 role I ) , where I denotes the identity matrix and we have  role = MLP role ( h X ; h ctxrole ) ,  role = Softplus(MLP role ( h X ; h ctxrole )) , where MLP(  ) and Softplus(  ) are multi-layer per-ceptron and approximation of ReLU function, respectively. (  ;  ) indicates concatenation operation. At training, the posterior distribution conditions on both role-specific utterances and the current translation, which contain rich role preference information. Therefore, the prior network can learn a role-tailored distribution by approaching the posterior network via KL divergence (Sohn et al., 2015): q  ( z role | X u , C roleX , Y u )  N (  (cid:48) role ,  (cid:48) 2 role I ) and {  (cid:48) role ,  (cid:48) role } are calculated as:  (cid:48) role = MLP role ( h X ; h ctxrole ; h Y ) ,  (cid:48) role = Softplus(MLP role ( h X ; h ctxrole ; h Y )) . Dialogue Coherence. To maintain the coherence in chat translation, we encode the entire source-language utterances and then generate a latent variable z dia  R d z . Similar to z role , we define its prior distribution as: p  ( z dia | X u , CX )  N (  dia ,  2 dia I ) and {  dia ,  dia } are calculated as:  dia = MLP dia ( h X ; h ctxX ) ,  dia = Softplus(MLP dia ( h X ; h ctxX )) . At training, the posterior distribution conditions on both the entire source-language utterances and the translation that provide a dialogue-level coherence clue, and is responsible for guiding the learning of the prior distribution. Specifically, we define the posterior distribution as: q  ( z dia | X u , CX , Y u )  N (  (cid:48) dia ,  (cid:48) 2 dia I ) , where  (cid:48) dia and  (cid:48) dia are calculated as:  (cid:48) dia = MLP dia ( h X ; h ctxX ; h Y ) ,  (cid:48) dia = Softplus(MLP dia ( h X ; h ctxX ; h Y )) . Translation Consistency. To keep the lexical choice of translation consistent with those of previous utterances, we encode the paired source-target utterances and then sample a latent variable z tra  R d z . We define its prior distribution as: p  ( z tra | X u , CX , CY )  N (  tra ,  2 tra I ) and {  tra ,  tra } are calculated as:  tra = MLP tra ( h X ; h ctxX ; h ctxY ) ,  tra = Softplus(MLP tra ( h X ; h ctxX ; h ctxY )) . and serves as learning of the prior distribution. Specifically, we define the posterior distribution as: q  ( z tra | X u , CX , CY , Y u )  N (  (cid:48) tra ,  (cid:48) 2 tra I ) , where  (cid:48) tra and  (cid:48) tra are calculated as:  (cid:48) tra = MLP tra ( h X ; h ctxX ; h ctxY ; h Y ) ,  (cid:48) tra = Softplus(MLP tra ( h X ; h ctxX ; h ctxY ; h Y )) . The decoder adopts a similar structure to the encoder, and each of N d decoder layers contains an additional cross-attention sub-layer ( CrossAtt ): s (cid:96)d = SelfAtt( h (cid:96)  1 d ) + h (cid:96)  1 d , h (cid:96)  1 d  R n  d , c (cid:96)d = CrossAtt( s (cid:96)d , h N e e ) + s (cid:96)d , s (cid:96)d  R n  d , h (cid:96)d = FFN( c (cid:96)d ) + c (cid:96)d , { c (cid:96)d , h (cid:96)d }  R n  d , where h (cid:96)d denotes the state of the (cid:96) -th decoder layer. As shown in Fig. 3, we obtain the latent variables { z role , z dia , z tra } either from the posterior distribution predicted by recognition networks (training process as the solid grey lines) or from prior distribution predicted by prior networks (inference process as the dashed red lines). Finally, we incorporate { z role , z dia , z tra } into the state of the top layer of the decoder with a projection layer: o t = Tanh( W p [ h N d d,t ; z role ; z dia ; z tra ] + b p ) , o t  R d , where W p  R d  ( d +3 d z ) and b p  R d are training parameters, h N d d,t is the hidden state at time-step t of the N d -th decoder layer. Then, o t is fed to a linear transformation and softmax layer to predict the probability distribution of the next target token: p t = Softmax( W o o t + b o ) , p t  R | V | , where W o  R | V | d and b o  R | V | are training parameters. We apply a two-stage training strategy (Zhang et al., 2018; Ma et al., 2020). Firstly, we train our model on large-scale sentence-level NMT data to minimize the cross-entropy objective: L (  ; X, Y ) =  N (cid:88) t =1 log p  ( y t | X, y 1: t  1 ) . Secondly, we fine-tune it on the chat translation data to maximize the following objective: J ( ,  ; X u , C roleX , CX , CY , Y u ) =  KL( q  ( z role | X u , C roleX , Y u ) (cid:107) p  ( z role | X u , C role X ))  KL( q  ( z dia | X u , CX , Y u ) (cid:107) p  ( z dia | X u , CX ))  KL( q  ( z tra | X u , CX , CY , Y u ) (cid:107) p  ( z tra | X u , CX , CY )) + E q  [log p  ( Y u | X u , z role , z dia , z tra )] . Datasets. We apply a two-stage training strategy, i.e. , firstly training on a large-scale sentence-level NMT corpus (WMT20 6 ) and then fine-tuning on chat translation corpus (BConTrasT (Farajian et al., 2020) 7 and BMELD). The details (WMT20 data and results of the first stage) are shown in Appendix A. BConTrasT. The dataset 8 is first provided by WMT 2020 Chat Translation Task (Farajian et al., 2020), which is translated from English into German and is based on the monolingual Taskmaster-1 corpus (Byrne et al., 2019). The conversations (originally in English) were first automatically translated into German and then manually post-edited by Unbabel editors, 9 who are native German speakers. Having the conversations in both languages allows us to simulate bilingual conversations in which one speaker, the customer, speaks in German and the other speaker, the agent, answers in English. BMELD. Similarly, based on the dialogue dataset in the MELD (originally in English) (Poria et al., 2019), 10 we firstly crawled the corresponding Chinese translations from this 11 and then manually post-edited them according to the dialogue history by native Chinese speakers, who are postgraduate students majoring in English. Finally, following (Farajian et al., 2020), we assume 50% speakers as Chinese speakers to keep data balance for Ch  En translations and build the bilingual MELD (BMELD). For the Chinese, we segment the sentence using Stanford CoreNLP toolkit 12 . Metrics. For fair comparison, we use the Sacre-BLEU 13 (Post, 2018) and v0.7.25 for TER (Snover 6 http://www.statmt.org/wmt20/translation-task.html 7 http://www.statmt.org/wmt20/chat-task.html 8 https://github.com/Unbabel/BConTrasT 9 www.unbabel.com 10 The MELD is a multimodal emotionLines dialogue dataset, each utterance of which corresponds to a video, voice, and text, and is annotated with detailed emotion and sentiment. 11 https://www.zimutiantang.com/ 12 https://stanfordnlp.github.io/CoreNLP/index.html 13 BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+ version.1.4.13 Dataset # Dialogues # Utterances Train Valid Test Train Valid Test En  De 550 78 78 7,629 1,040 1,133 De  En 550 78 78 6,216 862 967 En  Ch 1,036 108 274 5,560 567 1,466 Ch  En 1,036 108 274 4,427 517 1,135 Table 1: Statistics of chat translation data. et al., 2006) (the lower the better) with the statistical significance test (Koehn, 2004). For En  De, we report case-sensitive score following the WMT20 chat task (Farajian et al., 2020). For Ch  En, we report case-insensitive score. For En  Ch, we report the character-level BLEU score. For all experiments, we follow the Transformer-Base and Transformer-Big settings illustrated in (Vaswani et al., 2017). In Transformer-Base , we use 512 as hidden size ( i.e. , d ), 2048 as filter size and 8 heads in multi-head attention. In Transformer-Big , we use 1024 as hidden size, 4096 as filter size, and 16 heads in multi-head attention. All our Transformer models contain N e = 6 encoder layers and N d = 6 decoder layers and all models are trained using THUMT (Tan et al., 2020) framework. We conduct experiments on the validation set of En  De to select the hyperparameters of context length and latent dimension, which are then shared for all tasks. For the results and more details (other hyperparameters setting and average running time), please refer to Appendix B, C, and D. 5.3 Comparison Models Baseline NMT Models. Transformer (Vaswani et al., 2017): the de-facto NMT model that does not fine-tune on chat translation data. Transformer+FT: fine-tuning on the chat translation data after being pre-trained on sentence-level NMT corpus. Context-Aware NMT Models. Doc-Transformer+FT (Ma et al., 2020): a state-of-the-art document-level NMT model based on Transformer sharing the first encoder layer to incorporate the bilingual dialogue history. Dia-Transformer+FT (Maruf et al., 2018): using an additional RNN-based (Hochreiter and Schmidhuber, 1997) encoder to incorporate the mixed-language dialogue history, where we re-implement it based on Transformer and use another Transformer layer to introduce context. V-Transformer+FT (Zhang et al., 2016; McCarthy Models En  De De  En En  Ch Ch  En BLEU  TER  BLEU  TER  BLEU  TER  BLEU  TER  Baseline NMT models (Base) Transformer 40.02 42.5 48.38 33.4 21.40 72.4 18.52 59.1 Transformer+FT 58.43 26.7 59.57 26.2 25.22 62.8 21.59 56.7 Context-Aware NMT models (Base) Doc-Transformer+FT 58.15 27.1 59.46 25.7 24.76 63.4 20.61 59.8 Dia-Transformer+FT 58.33 26.8 59.09 26.2 24.96 63.7 20.49 60.1 V-Transformer+FT 58.74 26.3 58.67 27.0 26.82 60.6 21.86 56.3 Ours (Base) CPCC 60.13  25.4  61.05  24.9  27.55  60.1  22.50  55.7  Baseline NMT models (Big) Transformer 40.53 42.2 49.90 33.3 22.81 69.6 19.58 57.7 Transformer+FT 59.01 26.0 59.98 25.9 26.95 60.7 22.15 56.1 Context-Aware NMT models (Big) Doc-Transformer+FT 58.61 26.5 59.98 25.4 26.45 62.6 21.38 57.7 Dia-Transformer+FT 58.68 26.8 59.63 26.0 26.72 62.4 21.09 58.1 V-Transformer+FT 58.70 26.2 60.01 25.7 27.52 60.3 22.24 55.9 Ours (Big) CPCC 60.23  25.6  61.45  24.8  28.98  59.0  22.98  54.6  Table 2: Results on BConTrasT (En  De) and BMELD (En  Ch) in terms of BLEU (%) and TER (%). et al., 2020): the variational NMT model based on Transformer also sharing the first encoder layer to exploit the bilingual context for fair comparison. Overall, we separate the models into two parts in Tab. 2: the Base setting and the Big setting. In each part, we show the results of our re-implemented Transformer baselines, the context-aware NMT systems, and our approach on En  De and En  Ch. Results on En  De. Under the Base setting, CPCC substantially outperforms the baselines ( e.g. , Transformer+FT) by a large margin with 1.70  and 1.48  BLEU scores on En  De and De  En, respectively. On the TER, our CPCC achieves a significant improvement of 1.3 points in both language pairs. Under the Big setting, our CPCC also consistently boosts the performance in both directions ( i.e. , 1.22  and 1.47  BLEU scores, 0.4  and 1.1  TER scores), showing its effectiveness. Compared against the strong context-aware NMT systems (underlined results), our CPCC significantly surpasses them (about 1.39  1.59  BLEU scores and 0.6  0.9  TER scores) in both language directions under both Base and Big settings, demonstrating the superiority of our model. Results on En  Ch. We also conduct experiments on our self-collected data to validate the generalizability across languages in Tab. 2. Our CPCC presents remarkable BLEU improvements over the Transformer+FT by a large margin in two directions by 2.33  and 0.91  BLEU gains under the Base setting, respectively, and by 2.03  and 0.83  BLEU gains in both directions under the Big setting. These results suggest that CPCC consistently performs well across languages. Compared with strong context-aware NMT systems ( e.g. , V-Transformer+FT), our approach notably surpasses them in both language directions under both Base and Big settings, which shows the generalizability and superiority of our model. We conduct ablation studies to investigate how well each tailored latent variable of our model works. When removing latent variables listed in Tab. 3, we have the following findings. (1) All latent variables make substantial contributions to performance, proving the importance of modeling role preference, dialogue coherence, and translation consistency, which is consistent with our intuition that the properties should be beneficial to better translations (rows 1  3 vs. row 0). (2) Results of rows 4  7 show the combination effect of three latent variables, suggesting that the combination among three latent variables has a cumulative effect (rows 4  7 vs. rows 0  3). (3) Row 7 vs. row 0 shows that explicitly modeling the bilingual conversational characteristics significantly outperforms implicit modeling ( i.e. , just incorporating the dialogue history into encoders), which lacks the relevant information guidance. Following (Lapata and Barzilay, 2005; Xiong et al., 2019), we measure dialogue coherence as sentence similarity. Specifically, the representation of each sentence is the mean of the distributed vectors of its words, and the dialogue coherence between two sentences s 1 and s 2 is determined by the cosine similarity: sim ( s 1 , s 2 ) = cos ( f ( s 1 ) , f ( s 2 )) , f ( s i ) = 1 | s i | (cid:88) w  s i ( w ) , where w is the vector for word w . We use Word2Vec 14 (Mikolov et al., 2013) to learn the distributed vectors of words by training on the monolingual dialogue dataset: Taskmaster-1 (Byrne et al., 2019). And we set the dimensionality of word embeddings to 100. Tab. 4 shows the cosine similarity on the test set of De  En. It reveals that our model encouraged by tailor-made latent variables produces better coherence in chat translation than contrast systems. Inspired by (Bao et al., 2020; Farajian et al., 2020), we use four criteria for human evaluation: (1) Preference measures whether the translation preserves the role preference information; (2) Coherence denotes whether the translation is semantically coherent with the dialogue history; (3) Consistency measures whether the lexical choice of translation is consistent with the preceding utterances; (4) Fluency measures whether the translation is logically reasonable and grammatically correct. 14 https://code.google.com/archive/p/word2vec/ Models 1-th Pr. We firstly randomly sample 200 examples from the test set of Ch  En. Then, we assign each bilingual dialogue history and corresponding 6 generated translations to three human annotators without order, and ask them to evaluate whether each translation meets the criteria defined above. All annotators are postgraduate students and not involved in other parts of our experiments. Tab. 5 shows that our CPCC effectively alleviates the problem of role-irrelevant, incoherent and inconsistent translations compared with other models (significance test (Koehn, 2004), p < 0.05), indicating the superiority of our model. The inter-annotator agreement is 0.527, 0.491, 0.556 and 0.485 calculated by the Fleiss' kappa (Fleiss and Cohen, 1973), for preference, coherence, consistency and fluency, respectively, indicating Mod-erate Agreement for all four criteria. We also present some case studies in Appendix H. 7 Related Work Chat NMT. It only involves several researches due to the lack of human-annotated publicly available data (Farajian et al., 2020). Therefore, some existing work (Wang et al., 2016; Maruf et al., 2018; Zhang and Zhou, 2019; Rikters et al., 2020) mainly pays attention to designing methods to automatically construct the subtitles corpus, which may contain noisy bilingual utterances. Recently, Farajian et al. (2020) organize the WMT20 chat translation task and first provide a human post-edited corpus, where some teams investigate the effect of dialogue history and finally ensemble their models for higher ranks (Berard et al., 2020; Mohammed et al., 2020; Wang et al., 2020; Bao et al., 2020; Moghe et al., 2020). As a synchronizing study, Wang et al. (2021) use multitask learning to auto-correct the translation error, such as pronoun dropping, punctuation dropping, and typos. Unlike them, we focus on explicitly modeling role preference, dialogue coherence, and translation consistency with tailored latent variables to promote the translation quality. Context-Aware NMT. Chat NMT can be viewed as a special case of context-aware NMT, which has attracted many researchers (Gong et al., 2011; Jean et al., 2017; Wang et al., 2017b; Bawden et al., 2018; Miculicich et al., 2018; Kuang et al., 2018; Tu et al., 2018; Yang et al., 2019; Kang et al., 2020; Li et al., 2020; Ma et al., 2020) to extend the encoder or decoder for exploring the context im-pact on translation quality. Although these models can be directly applied to chat translation, they cannot explicitly capture the bilingual conversational characteristics and thus lead to unsatisfactory translations (Moghe et al., 2020). Different from these studies, we focus on explicitly modeling these bilingual conversational characteristics via CVAE for better translations. Conditional Variational Auto-Encoder. CVAE has verified its superiority in many fields (Sohn et al., 2015). In NMT, Zhang et al. (2016) and Su et al. (2018) extend CVAE to capture the global/local information of source sentence for better results. McCarthy et al. (2020) focus on addressing the posterior collapse with mutual information. Besides, some studies use CVAE to model the correlations between image and text for multimodal NMT (Toyama et al., 2016; Calixto et al., 2019). Although the CVAE has been widely used in NLP tasks, its adaption and utilization to chat translation for modeling inherent bilingual conversational characteristics are non-trivial, and to the best of our knowledge, has never been investigated before. We propose to model bilingual conversational characteristics through tailored latent variables for neural chat translation. Experiments on En  De and En  Ch directions show that our model notably improves translation quality on both BLEU and TER metrics, showing its superiority and generalizability. Human evaluation further verifies that our model yields role-specific, coherent, and consistent translations by incorporating tailored latent variables into NMT. Moreover, we contribute a new bilingual dialogue data (BMELD, En  Ch) with manual translations to the research community. In the future, we would like to explore the effect of multimodality and emotion on chat translation, which has been well studied in dialogue field (Liang et al., 2020). The research work descried in this paper has been supported by the National Key R&D Program of China (2020AAA0108001) and the National Nature Science Foundation of China (No. 61976015, 61976016, 61876198 and 61370130). The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to improve this paper.
abstain abstain objective method abstain method result abstain abstain abstain abstain abstain other abstain abstain other other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective abstain method method method result method abstain method objective objective result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain method method abstain abstain method abstain method abstain abstain method method abstain abstain method abstain method abstain abstain method abstain method method abstain abstain other abstain method method abstain method abstain method method abstain method abstain other abstain abstain method abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain method abstain abstain abstain method other method abstain abstain abstain abstain method other method method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain other abstain method abstain abstain abstain abstain abstain abstain objective result abstain objective abstain other other