Lexically constrained neural machine translation (NMT), which controls the generation of NMT models with pre-specified constraints, is important in many practical scenarios. Due to the representation gap between discrete constraints and continuous vectors in NMT models, most existing works choose to construct synthetic data or modify the decoding algorithm to impose lexical constraints, treating the NMT model as a black box. In this work, we propose to open this black box by directly integrating the constraints into NMT models. Specifically, we vectorize source and target constraints into continuous keys and values, which can be utilized by the attention modules of NMT models. The proposed integration method is based on the assumption that the correspondence between keys and values in attention modules is naturally suitable for modeling constraint pairs. Experimental results show that our method consistently outperforms several representative baselines on four language pairs, demonstrating the superiority of integrating vectorized lexical constraints. 1 1 Introduction Controlling the lexical choice of the translation is important in a wide range of settings, such as interactive machine translation (Koehn, 2009), entity translation (Li et al., 2018), and translation in safety-critical domains (Wang et al., 2020). However, different from the case of statistical machine translation (Koehn et al., 2007), it is non-trivial to directly integrate discrete lexical constraints into neural machine translation (NMT) models (Bah-danau et al., 2015; Vaswani et al., 2017), whose hidden states are all continuous vectors that are difficult for humans to understand. In accordance with this problem, one branch of studies directs its attention to designing adCorrespondence to: Yang Liu. vanced decoding algorithms (Hokamp and Liu, 2017; Hasler et al., 2018; Post and Vilar, 2018) to impose hard constraints and leave NMT models unchanged. For instance, Hu et al. (2019) propose a vectorized dynamic beam allocation (VDBA) algorithm, which devotes part of the beam to candidates that have met some constraints. Although this kind of method can guarantee the presence of target constraints in the output, they are found to potentially result in poor translation quality (Chen et al., 2021b; Zhang et al., 2021), such as repeated translation or source phrase omission. Another branch of works proposes to learn constraint-aware NMT models through data augmentation. They construct synthetic data by replacing source constraints with their target-language correspondents (Song et al., 2019) or appending target constraints right after the corresponding source phrases (Dinu et al., 2019). During inference, the input sentence is edited in advance and then provided to the NMT model. The major drawback of data augmentation based methods is that they may suffer from a low success rate of generating target constraints in some cases, indicating that only adjusting the training data is sub-optimal for lexical constrained translation (Chen et al., 2021b). To make NMT models better learn from and cope with lexical constraints, we propose to lever-7063 age attention modules (Bahdanau et al., 2015) to explicitly integrate vectorized lexical constraints. As illustrated in Figure 1, we use vectorized source constraints as additional keys and vectorized target constraints as additional values. Intuitively, the additional keys are used to estimate the relevance between the current query and the source phrases while the additional values are used to integrate the information of the target phrases. In this way, each revised attention is aware of the guidance to translate which source phrase into what target phrase . Experiments show that our method can significantly improve the ability of NMT models to translate with constraints, indicating that the correspondence between attention keys and values is suitable for modeling constraint pairs. Inspired by recent progress in controlled text generation (Dathathri et al., 2020; Pascual et al., 2021), we also introduce a plug-in to the output layer that can further improve the success rate of generating constrained tokens. We conduct experiments on four language pairs and find that our model can consistently outperform several representative baselines. Training The goal of machine translation is to translate a source-language sentence x = x 1 . . . x | x | into a target-language sentence y = y 1 . . . y | y | . We use P ( y | x ;  ) to denote an NMT model (Vaswani et al., 2017) parameterized by  . Modern NMT models are usually trained by maximum likelihood estimation (Bahdanau et al., 2015; Vaswani et al., 2017), where the log-likelihood is defined as log P ( y | x ;  ) = | y | (cid:88) t =1 log P ( y t | y <t , x ;  ) , (1) in which y <t is a partial translation. Inference The inference of NMT models can be divided into two sub-processes:  probability estimation : the model estimates the token-level probability distribution for each partial hypothesis within the beam;  candidate selection : the decoding algorithm selects some candidates based on the probability estimated by the NMT model. These two sub-processes are performed alternatively until reaching the maximum length or generating the end-of-sentence token. This section explains how we integrate lexical constraints into NMT models. Section 3.1 illustrates the way we encode discrete constraints into continuous vectors, Section 3.2 details how we integrate the vectorized constraints into NMT models, and Section 3.3 describes our training strategy. Let s = s (1) , . . . , s ( N ) be the source constraints and t = t (1) , . . . , t ( N ) be the target constraints. Given a constraint pair  s ( n ) , t ( n )  , lexically constrained translation requires that the system must translate the source phrase s ( n ) into the target phrase t ( n ) . Since the inner states of NMT models are all continuous vectors rather than discrete tokens, we need to vectorize the constraints before integrating them into NMT models. For the n -th constraint pair  s ( n ) , t ( n )  , let | s ( n ) | and | t ( n ) | be the lengths of s ( n ) and t ( n ) , respectively. We use S ( n ) k  R d  1 to denote the vector representation of the k -th token in s ( n ) , which is the sum of word embedding and positional embedding (Vaswani et al., 2017). Therefore, the matrix representation of s ( n ) is given by: S ( n ) = (cid:104) S ( n ) 1 ; . . . ; S ( n ) | s ( n ) | (cid:105) , (2) where S ( n )  R d | s ( n ) | is the concatenation of all vector representations of tokens in s ( n ) . Similarly, the matrix representation of the target constraint t ( n ) is T ( n )  R d | t ( n ) | . Note that the positional embedding for each constraint is calculated independently, which is also independent of the positional embeddings of the source sentence x and the target sentence y . We adopt Transformer (Vaswani et al., 2017) as our NMT model, which is nowadays one of the most popular and effective NMT models (Liu et al., 2020). Typically, a Transformer consists of an encoder, a decoder, and an output layer, of which the encoder and decoder map discrete tokens into vectorized representations and the output layer converts such representations into token-level probability distributions. We propose to utilize the attention modules to integrate the constraints into the encoder and decoder and use a plug-in module to integrate constraints into the output layer. We change the formal representation of our model 7064 Encoder Decoder Constraints Q y 1 Q y 3 Q y 5 Q y 4 Q y 2 K x 2 K x 1 K x 3 K x 4 K x 5 K s (1)1 c K s (2)1 c V x 2 V x 1 V x 3 V x 4 V x 5 V t (1)1 c V t (2)1 c from P ( y | x ;  ) to P ( y | x , s , t ;  ) to indicate that the model explicitly considers lexical constraints when estimating probability. Constraint-Related Keys and Values We propose to map source and target constraints into additional keys and values, which are called constraint-related keys and values , in order to distinguish from the original keys and values in vanilla attention modules. In practice, source and target constraints may have different lengths and they are usually not monotonically aligned (Du et al., 2021), making it challenging to directly convert the constraints into keys and values. To fix this problem, We adopt a multi-head attention layer (Vaswani et al., 2017) to align the bilingual constraints. The constraint-related keys and values for the n -th constraint pair are given by K ( n ) c = S ( n ) , V ( n ) c = attn (cid:16) S ( n ) , T ( n ) , T ( n ) (cid:17) , (3) where K ( n ) c  R d | s ( n ) | and V ( n ) c  R d | s ( n ) | . attn( Q , K , V ) denotes the multi-head attention function. Note that the resulting K ( n ) c and V ( n ) c are of the same shape. V ( n ) c can be seen as a redistributed version of the representation of target constraints. The constraint-related keys and values of each constraint pair are calculated separately and then concatenated together: K c = [ K (1) c ; . . . ; K ( N ) c ] , V c = [ V (1) c ; . . . ; V ( N ) c ] , (4) where K c  R d | s | and V c  R d | s | . | s | is the total length of all the N source constraints. Integration into the Encoder The encoder of Transformer is a stack of I identical layers, each layer contains a self-attention module to learn context-aware representations. For the i -th layer, the self-attention module can be represented as attn (cid:16) H ( i  1) enc , H ( i  1) enc , H ( i  1) enc (cid:17) , (5) where H ( i  1) enc  R d | x | is the output of the ( i  1) th layer, and H (0)enc is initialized as the sum of word embedding and positional embedding (Vaswani et al., 2017). For different layers, H ( i  1) enc may lay in various manifolds, containing different levels of information (Voita et al., 2019). Therefore, we should adapt the constraint-related keys and values for each layer before the integration. We use a two-layer adaptation network to do this: K ( i ) c4enc = [adapt( K c ); H ( i  1) enc ] , V ( i ) c4enc = [adapt( V c ); H ( i  1) enc ] , (6) where adapt(  ) denotes the adaptation network, which consists of two linear transformations with shape d  d and a ReLU activation in between. The adaptation networks across all layers are independent of each other. K ( i ) c4enc  R d  ( | s | + | x | ) and V ( i ) c4enc  R d  ( | s | + | x | ) are the constraint-aware keys and values for the i -th encoder layer, respectively. The vanilla self-attention module illustrated in Eq. (5) is revised into the following form: attn (cid:16) H ( i  1) enc , K ( i ) c4enc , V ( i ) c4enc (cid:17) . Integration into the Decoder The integration into the decoder is similar to that into the encoder, the major difference is that we use the cross-attention module to model constraints for the decoder. The decoder of the Transformer is a stack of J identical layers, each of which is composed of a self-attention, a cross-attention, and a feed-forward module. We integrate vectorized constraints into the cross-attention module for the decoder. Formally, the vanilla cross-attention is given by attn (cid:16) S ( j ) dec , H ( I ) enc , H ( I ) enc (cid:17) , (8) where S ( j ) dec  R d | y | is the output of the self-attention module in the j -th decoder layer, and H ( I ) enc  R d | x | is the output of the last encoder layer. We adapt the constraint-related keys and values to match the manifold in the j -th decoder layer: K ( j ) c4dec = [adapt( K c ); H ( I ) enc ] , V ( j ) c4dec = [adapt( V c ); H ( I ) enc ] . (9) Then we revise the vanilla cross-attention (Eq. (8)) into the following form: attn (cid:16) S ( j ) dec , K ( j ) c4dec , V ( j ) c4dec (cid:17) . (10) Figure 2c plots an example of the integration into the decoder cross-attention. Integration into the Output Layer In vanilla Transformer, an output layer is employed to convert the output of the last decoder layer into token-level probabilities. Let h t  R d  1 be the decoder output at the t -th time step, the output probability of the Transformer model is defined as P model ( y | y <t , x , s , t ;  ) = softmax (cid:16) h  t W (cid:17) , (11) where W  R d |V| is the output embedding matrix and |V| is the vocabulary size. Inspired by the plug-and-play method (Pascual et al., 2021) in the field of controlled text generation (Dathathri et al., 2020; Pascual et al., 2021), we introduce an additional probability distribution over the vocabulary to better generate constrained tokens: P plug ( y | y <t , x , s , t ;  ) =   0 y /  t max (cid:18) 0 , cos (cid:18) w y | w y | , h t | h t | (cid:19)(cid:19) y  t , (12) where w y  R d  1 is the word embedding of token y and t is the sequence of all the target-side constrained tokens. We also use a gating sub-layer to control the strength of the additional probability: g ( y, h t ) = sigmoid (cid:16) tanh (cid:16)(cid:104) w  y W 1 ; h  t W 2 (cid:105)(cid:17) W 3 (cid:17) , (13) where W 1  R d  d , W 2  R d  d , and W 3  R 2 d  1 are three trainable linear transformations. The final output probability is given by P ( y | y <t , x , s , t ;  ) = (1  g ( y, h t )) P model ( y | y <t , x , s , t ;  ) + g ( y, h t ) P plug ( y | y <t , x , s , t ;  ) . (14) 3.3 Training and Inference Training The proposed constraint-aware NMT model should not only generate pre-specified constraints but also maintain or improve the translation quality compared with vanilla NMT models. We thus propose to distinguish between constraint tokens and constraint-unrelated tokens during training. Formally, the training objective is given by L ( y | x , s , t ;  ) =  (cid:88) y t  y  t log P ( y t | y <t , x , s , t ;  ) +  (cid:88) y t  y \ t log P ( y t | y <t , x , s , t ;  ) , (15) where  and  are hyperparameters to balance the learning of constraint generation and translation. is a set of original vanilla model parameters and  c is a set of newly-introduced parameters that are used to vectorize and integrate lexical constraints. 2 Since  c is significantly smaller than  v , it requires much less training iterations. Therefore, we adopt the strategy of two-stage training (Tu et al., 2018; Zhang et al., 2018) for model optimization. Specifically, we optimize  v using the standard NMT training objective (Bahdanau et al., 2015; Vaswani et al., 2017) at the first stage and then learn the whole model  at the second stage. The second stage is significantly shorter than the first stage, we will give more details in Section 4.1. Inference As discussed in Section 2, the inference process is composed of two sub-processes: probability estimation and candidate selection. In this work, we aim to improve the probability estimation sub-process and our method is orthogonal to constrained decoding algorithms (Hokamp and Liu, 2017; Post and Vilar, 2018; Hu et al., 2019), which instead focus on candidate selection. Therefore, we can employ not only beam search but also constrained decoding algorithms at inference time. We use VDBA (Hu et al., 2019) as the default constrained decoding algorithm, which supports batched inputs and is significantly faster than most other counterparts (Hokamp and Liu, 2017; Post and Vilar, 2018; Hasler et al., 2018). Training Data In this work, we conduct experiments on Chinese  English (Zh  En) and German  English (De  En) translation tasks. For Zh  En, the training set contains 1.25M sentence pairs from LDC 3 . For De  En, the training set is from the WMT 2014 German  English translation task, which consists of 4.47M sentence pairs. We apply BPE (Sennrich et al., 2016b) with 32K joint merge operations for both Zh  En and De  En. Evaluation Data Following Chen et al. (2021b), we evaluate our approach on the test sets with human-annotated alignments, which are widely used in related studies (Chen et al., 2020b, 2021a). 2  c includes parameters of the attention presented in Eq (3), the adaptation networks described in Eq (6) and (9), and the gating sub-layer illustrated in Eq (13). We find the alignment test sets have significant overlaps with the corresponding training sets, which is not explicitly stated in previous works. In this work, we remove the training examples that are covered by the alignment test sets. For Zh  En, we use the alignment datasets from Liu et al. (2005) 4 , in which the validation and test sets both contain 450 sentence pairs. For De  En, we use the alignment dataset from Zenkel et al. (2020) 5 as the test set, which consists of 508 sentence pairs. Since there is no human-annotated alignment validation sets for De  En, we use fast-align 6 to annotate the newstest 2013 as the validation set for De  En. Lexical Constraints In real-world applications, lexical constraints are usually provided by human translators. We follow Chen et al. (2021b) to simulate the practical scenario by sampling constraints from the phrase pairs that are extracted from parallel data using alignments. The script for phrase pair extraction is publicly available. 7 For the validation and test sets of Zh  En and the test set of De  En, we use human-annotated alignments to extract phrase pairs. For the training corpora in both Zh  En and De  En, we use fast-align to firstly learn an alignment model and then use the model to automatically annotate the alignments. The validation set of De  En is also annotated by the alignment model learned on the corresponding training corpus. We use the same strategy as Chen et al. (2021b) to sample constraints from the extracted phrase pairs. More concretely, the number of constraints in each sentence is up to 3. The length of each constrained phrase is uniformly sampled among 1 and 3. For each sentence pair, all the constraint pairs are shuffled and then supplied to the model in an unordered manner. Model Configuration We use the base setting (Vaswani et al., 2017) for our model. Specifically, the hidden size d is 512 and the depths of both the encoder and the decoder are 6. Each multihead attention module has 8 individual attention heads. Since our method introduces additional parameters, we use a larger model with an 8-layer encoder and an 8-layer decoder to assimilate the 4 http://nlp.csai.tsinghua.edu.cn/~ly/ systems/TsinghuaAligner/TsinghuaAligner.html 5 https://github.com/lilt/ alignment-scripts 6 https://github.com/clab/fast_align 7 https://github.com/ghchen18/cdalign/ blob/main/scripts/extract_phrase.py 7067 Method BLEU CSR (%) Z  E E  Z D  E E  D Avg. parameter count for the baselines. For Zh  En, we optimize  v for 50K iterations at the first stage and then optimize  for 10K iterations at the second stage. For a fair comparison, we train the baselines for 60K iterations in total. For De  En, we optimize  v for 90K iterations then optimize  for 10K iterations. The baselines are trained for 100K iterations. All the involved models are optimized by Adam (Kingma and Ba, 2015), with  1 = 0 . 9 ,  2 = 0 . 98 and  = 10  9 . The dropout rate is set to 0.3 for Zh  En and 0.1 for De  En. Label smoothing is employed and the smoothing penalty is set to 0.1 for all language pairs. We use the same learning rate schedule as Vaswani et al. (2017). All models are trained on 4 NVIDIA V100 GPUs and evaluated on 1 NVIDIA V100 GPU. During training, each mini batch contains roughly 32K tokens in total across all GPUs. We set the values of  and  based on the results on the validation set. Specifically, for models using VDBA, we set  =  = 0 . 5 , while for models using beam search, we set  = 0 . 8 and  = 0 . 2 . The beam size is set to 4 during inference. Baselines We compare our approach with three representative baselines:  VDBA (Hu et al., 2019): dynamically devoting part of the beam for constraint-related hypotheses at inference time;  Replace (Song et al., 2019): directly replacing source constraints in the training data with their corresponding target constraints. The model is also improved with pointer network;  CDAlign (Chen et al., 2021b): explicitly using an alignment model to decide the position to insert target constraints during inference. Evaluation Metrics We evaluate the involved methods using the following two metrics:  BLEU : we use sacreBLEU 8 (Post, 2018) to report the BLEU score;  Copying Success Rate (CSR) : We follow Chen et al. (2021b) to use the percentage of constraints that are successfully generated in the translation as the CSR, which is calculated at word level after removing the BPE separator. We use compare-mt (Neubig et al., 2019) for significance testing, with bootstrap = 1000 and prob _ thresh = 0 . 05 . Table 1 shows the results of lexically constrained translation on test sets of all four translation tasks. All the investigated methods can effectively improve the CSR over the vanilla Transformer. The CSR of VDBA on Zh  En is not 100.0% for the reason that some target constraints contain out-of-vocabulary tokens. Replace (Song et al., 2019) achieves better BLEU scores on three translation directions (i.e., Zh  En and En  De) than VDBA, but its CSR is much lower. CDAlign (Chen et al., 2021b) also performs better than Replace on average regarding CSR. Our method consistently outperforms all the three baselines across the four translation directions in terms of BLEU, demonstrating the necessity of integrating vectorized constraints into NMT models. Decoding with VDBA, we also achieve the highest CSR. To disentangle the effect of integrating vectorized constraints and 8 Signature for Zh  En, De  En, and En  De: nrefs:1 | case:mixed | eff:no | tok:13a | smooth:exp | version:2.0.0. Signature for En  Zh: nrefs:1 | case:mixed | eff:no | tok:zh | smooth:exp | version:2.0.0. VDBA, we also report the result of our model using beam search in Table 2. Decoding with beam search, our model can also achieve a better BLEU score than the baselines and the CSR is higher than both Replace and CDAlign on average. We investigate the effect of different components through an ablation study, the results are shown in Table 3. We find that only integrating lexical constraints into attention can significantly improve the CSR over the vanilla model (91.5% vs. 25.5%), which is consistent with our motivation that the correspondence between keys and values is naturally suitable for modeling the relation between source and target constraints. Plugging target constraints into the output layer can further improve the performance, but the output plug-in itself can only generate 61.9% of constraints. When decoding with VDBA, combining both the two types of integration achieves the best BLEU score, indicating that every component is important for the model to translate with constraints. Task Description and Data Preparation An interesting application of lexically constrained machine translation is code-switched translation, of which the output contains terms across different (a) BLEU score on code-switched test sets. languages. Figure 1 shows an example of code-switched translation, where the output Chinese sentence should include the English token "Beatles". Code-switched machine translation is important in many scenarios, such as entity translation (Li et al., 2018) and the translation of sentences containing product prices or web URLs (Chen et al., 2021b). In this work, we evaluate the performance of several approaches on code-switched machine translation. The parallel data and extracted constraint pairs for each language pair are the same as those used in the lexically constrained translation task. To construct the training and evaluation data for code-switched translation, we randomly replace 50% of the target constraints with their corresponding source constraints. The target sentence is also switched if it contains switched target constraints. Results Table 4 gives the results of the code-switched translation task. The CSR of Replace (Song et al., 2019) is lower than 50% across all the four translation directions, indicating that simply replacing the training data can not handle the code-switched translation. A potential reason is that it is difficult for the NMT model to decide whether to translate or copy some source phrases in the input sentence. Surprisingly, VDBA, CDAlign, 7069 Method Batch Size (# Sent.) 1 128 Vanilla 1.0  43.2  VDBA 0.5  2.1  Replace 0.9  40.5  CDAlign 0.7  n/a Ours 0.5  2.3  w/o VDBA 0.9  39.2  Table 5: Inference speed with different batch sizes. and our method all perform well in this scenario, and our method outperforms the two baselines. These results suggest the capability of our method to cope with flexible types of lexical constraints. We report the inference speed of each involved approach in Table 5. The speed of Replace is close to that of the vanilla Transformer, but its CSR is much lower than other methods. Since the open-sourced implementation of CDAlign 9 does not support batched decoding, we compare our method with CDAlign with batch _ size = 1 . The speed of our method using beam search is faster than that of CDAlign (0.9  vs. 0.7  ). When provided with batched inputs, our method can slightly speed up VDBA (2.3  vs. 2.1  ). A potential reason is that the probability estimated by our model is more closely related to the correctness of the candidates, making target constraints easier to find. Wang et al. (2020) to investigate the gap between the probability and the correctness of model outputs, which is measured by the inference expected calibration error (ECE). As shown in Table 6, the inference ECE of our method is much lower than that of the vanilla model, indicating that the probability of our model is more accurate than vanilla models. To better understand the calibration of our model and the baseline model, we also estimate the average probability of all the predicted tokens and the constrained tokens. The results show that our model assigns higher probabilities to constrained tokens, which are already known to be correct. To address the concern that the proposed model may only memorize the constraints seen in the training set, we calculate the overlap ratio of constraints between training and test sets. As shown in Table 7, we find that only 35.6% of the test constraints are seen in the training data, while the CSR of our model decoding with beam search is 94.4%. The results indicate that our method extrapolates well to constraints unseen during training. Table 8 shows some example translations of different methods. We find Replace tends to omit some constraints. Although VDBA and CDAlign can successfully generate constrained tokens, the translation quality of the two methods is not satisfying. Our result not only contains constrained tokens but also maintains the translation quality compared with the unconstrained model, confirming the necessity of integrating vectorized constraints into NMT models. One line of approaches to lexically constrained NMT focuses on designing advanced decoding algorithms (Hasler et al., 2018). Hokamp and Liu (2017) propose grid beam search (GBS), which enforces target constraints to appear in the output by 7070 Constraints Zielsetzung  objectives , Fiorella  Fiorella Source Mit der Zielsetzung des Berichtes von Fiorella Ghilardotti allerdings sind wir einverstanden . enumerating constraints at each decoding step. The beam size required by GBS varies with the number of constraints. Post and Vilar (2018) propose dynamic beam allocation (DBA) to fix the problem of varying beam size for GBS, which is then extended by Hu et al. (2019) into VDBA that supports batched decoding. There are also some other constrained decoding algorithms that leverage word alignments to impose constraints (Song et al., 2020; Chen et al., 2021b). Although the alignment-based decoding methods are faster than VDBA, they may be negatively affected by noisy alignments, resulting in low CSR. Recently, Susanto et al. (2020) adopt Levenshtein Transformer (Gu et al., 2019) to insert target constraints in a non-autoregressive manner, for which the constraints must be provided with the same order as that in the reference. Another branch of studies proposes to edit the training data to induce constraints (Sennrich et al., 2016a). Song et al. (2019) directly replace source constraints with their target translations and Dinu et al. (2019) insert target constraints into the source sentence without removing source constraints. Similarly, Chen et al. (2020a) propose to append target constraints after the source sentence. In this work, we propose to integrate vectorized lexical constraints into NMT models. Our work is orthogonal to both constrained decoding and constraint-oriented data augmentation. A similar work to us is that Li et al. (2020) propose to use continuous memory to store only the target constraint, which is then integrated into NMT models through the decoder self-attention. However, Li et al. (2020) did not exploit the correspondence between keys and values to model both source and target constraints. Recent years have witnessed rapid progress in controlled text generation. Dathathri et al. (2020) propose to use the gradients of a discriminator to control a pre-trained language model to generate towards a specific topic. Liu et al. (2021) propose a decoding-time method that employs experts to control the generation of pre-trained language models. We borrow the idea presented in Pascual et al. (2021) to insert a plug-in into the output layer. The difference between our plug-in network and Pascual et al. (2021) is that we use an input-dependent gate to control the effect of the plugged probability. In this work, we propose to vectorize and integrate lexical constraints into NMT models. Our basic idea is to use the correspondence between keys and values in attention modules to model constraint pairs. Experiments show that our approach can outperform several representative baselines across four different translation directions. In the future, we plan to vectorize other attributes, such as the topic, the style, and the sentiment, to better control the generation of NMT models. This work is supported by the National Key R&D Program of China (No. 2018YFB1005103), the National Natural Science Foundation of China (No. 61925601, No. 62006138), and the Tencent AI Lab Rhino-Bird Focused Research Program (No. JR202031). We sincerely thank Guanhua Chen and Chi Chen for their constructive advice on technical details, and all the reviewers for their valuable and insightful comments.
abstain abstain objective method abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain result result result abstain method other abstain abstain method method abstain abstain method abstain method abstain abstain abstain method abstain method method abstain method abstain method other abstain abstain abstain other abstain abstain other abstain method method abstain abstain abstain abstain method abstain method other method method abstain abstain abstain abstain other method method abstain abstain method other abstain abstain method method method abstain method method method method abstain abstain method method abstain method method method method method abstain method abstain method method abstain method abstain abstain abstain method abstain abstain other abstain method method method abstain abstain abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain method method method result abstain result method result abstain result result abstain result abstain result other other other other other other other other other other other objective abstain abstain other other other other method method objective method result method other other