Prompting has recently been shown as a promising approach for applying pre-trained language models to perform downstream tasks. We present Multi-Stage Prompting, a simple and automatic approach for leveraging pre-trained language models to translation tasks. To better mitigate the discrepancy between pre-training and translation, MSP divides the translation process via pre-trained language models into multiple separate stages: the encoding stage, the re-encoding stage, and the decoding stage. During each stage, we independently apply different continuous prompts for allowing pre-trained language models better shift to translation tasks. We conduct extensive experiments on three translation tasks. Experiments show that our method can significantly improve the translation performance of pre-trained language models. 1 1 Introduction Prompting (Brown et al., 2020; Lester et al., 2021), which refers to the approach of generating task-specific outputs from language models (LMs) by conditioning on extra information (known as prompts ), has emerged as a new way of using LMs to perform natural language processing (NLP) tasks (Gao et al., 2020; Liu et al., 2021). While being efficient in parameters (Lester et al., 2021), prompting can enable mixed-task inference, which is not possible for other related approaches like finetuning or adapter-based tuning (Li and Liang, 2021; Lester et al., 2021). Prompting also opens the possibility of using a single pre-trained LM to perform all NLP tasks (Liu et al., 2021). and Knowles, 2017). While neural machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) is the current de facto approach for machine translation, using pre-trained LMs as translators via prompting is appealing in several aspects. For example, for the method described in this paper, supporting a new translation direction with a pre-trained LM occupies disk spaces below 20M, which is much smaller than training a separate neural machine translation model, where the model size is typically larger than 60M per language pair for the Transformer architecture. 2 Furthermore, the pre-trained LM also retains the ability to perform other downstream tasks, which is an important characteristic that has not been validated available on neural machine translation models. However, it is challenging to leverage pre-trained LMs to translation tasks via prompting. First, finding an appropriate prompt for a translation task is not trivial and requires specific designs (Brown et al., 2020; Gao et al., 2020; Li and Liang, 2021; Lester et al., 2021). Second, the prompting method with a single prompt may be sub-optimal for steering pre-trained LMs to translation tasks, as there is a clear discrepancy between the objectives of translation and pre-training. Translation imposes strict semantic equivalence and language space constraint, in which a source sentence must translate to a semantically equivalent sentence in the target language space. As the objective of pretraining is usually to reconstruct parts of the input sentence (Radford et al., 2018; Devlin et al., 2019), the generation of a pre-trained LM conditioned on a source sentence will likely be in the source language space with non-equivalent semantics. Therefore, using a single prompt to guide the LM for mitigating both the semantic and language gap is likely to be sub-optimal. Third, prevalent 2 Assume using the transformer-base setting with a vocabulary size of 32K. generative LMs such as GPTs use a decoder-only architecture (Radford et al., 2018), which is unidirectional and may be sub-optimal for encoding source sentences (Devlin et al., 2019). While re-cent works in prompting like prefix-tuning (Li and Liang, 2021) or prompt tuning (Lester et al., 2021) alleviate the first challenge by introducing differentiable continuous prompts, the last two challenges remain to be addressed. In this paper, we present Multi-Stage Prompting (MSP) for addressing the challenges of steering pre-trained language models to translation tasks. MSP encapsulates the idea of breaking translation tasks into simpler consecutive stages, allowing the pre-trained LM to learn smoother transitions to translation tasks by providing different prompts at different stages. For GPT-style pre-trained LMs, we design a three-stage prompting scheme for modeling the translation process, which consists of an encoding stage , a re-encoding stage , and a decoding stage . Specifically, the pre-trained LM focuses on learning source representations at the encoding stage and learns refined bidirectional representations by re-encoding source sentences at the re-encoding stage. Therefore, the LM can produce better translations with refined source representations at the decoding stage. Following prefix-tuning (Li and Liang, 2021) and prompt tuning (Lester et al., 2021), we use independent trainable continuous prompts at different stages, which are learned through back-propagation. The difference between basic (single-stage) prompting and multi-stage prompting is illustrated in Figure 1. We demonstrate the effectiveness of our method with a multilingual GPT (mGPT) model on Romanian-English, English-German, and English-Chinese translation tasks. Experiments verify that compared with prompt tuning or prefix-tuning, MSP can significantly improve the translation performance of pre-trained LMs. Our method improves the translation performance of pre-trained language models via prompt tuning and prefix-tuning by 18.6 and 4.1 BLEU points on average over the three translation tasks, respectively, suggesting that MSP is a more effective prompting method for translation tasks. Prompting is an approach of using an LM to perform downstream tasks by adding extra information for the LM to condition during its generation (Lester et al., 2021). This extra information, also known as a prompt , plays an important role in prompting methods and is often prepended to LM's input for better control of its generation. Depending on the form of prompts, prompting methods can be divided into two categories: using textual prompts or using continuous prompts. Textual prompts are typically composed of natural language tokens. As a representative approach of textual prompts, Brown et al. (2020) use manually designed prompts to steer GPT-3's generation. A typical prompt used in GPT-3 consists of a task description and a few task-specific examples. Gao et al. (2020) and Shin et al. (2020) propose different automatic methods to generate textual prompts. Textual prompts are typically understandable by humans. However, Shin et al. (2020) indicate that automatically generated textual prompts may lack interpretability. Continuous prompts, which consist of a sequence of continuous vectors, have gained increasing popularity recently. For example, in (Li and Liang, 2021), the continuous prompts consist of a sequence of key-value pairs (also called prefixes). Lester et al. (2021) propose a simplified version of continuous prompts, which consists of virtual 6132 tokens that are only added to the embedding layer. Compared with textual prompts, using continuous prompts is generally more powerful but less interpretable (Lester et al., 2021). In this paper, we use GPT (Radford et al., 2018, 2019; Brown et al., 2020) as the backbone LM for machine translation tasks. GPTs are a series of causal language models based on the Transformer architecture (Vaswani et al., 2017). To be more suitable for translation tasks that involve multiple languages, we introduce a multilingual GPT (mGPT) model instead of using a standard GPT-2 model. 3 The main difference between mGPT and GPT-2 is the training data. mGPT is trained on the mC4 dataset (Xue et al., 2021), which is a multilingual dataset covering over 101 languages. For further details about mGPT, please refer to Appendix A.1. Let z = [ z 1 , . . . , z n ] be a sequence of tokens, mGPT uses an autoregressive Transformer network to model the conditional probability P ( z t | z <t ) , where t  [1 , n ] and z <t = [ z 1 , . . . , z t  1 ] . We use f LM ( z , H ;  ) to denote the Transformer network, where z is a word embedding, H is a sequence of past activations, and  denotes the parameters of the Transformer network. Initially, the inputs to the Transformer network are z 1 and H 0 , where H 0 is an empty sequence. The Transformer network produces two outputs: the final output g 1  R d and the activation h 1  R 2 N  d , 4 where d denotes the hidden size of the Transformer network and N is the number of layers of the Transformer network. For subsequent inputs z t and H t  1 , where H t  1 = [ h 1 , . . . , h t  1 ] , the computation is formally described as g t , h t = f LM ( e z t , H t  1 ) , (1) where e z t denotes the word embedding of z t . To make the notation simpler, we use the following equation to denote the repeated application of f LM over a sequence z i : j = [ z i , . . . , z j ] given past activations A : G i : j , H i : j = f LM ( Z i : j , A ) , (2) where Z i : j = [ e z i , . . . , e z j ] , G i : j = [ g i , . . . , g j ] , and H i : j = [ h i , . . . , h j ] . 3 We release our checkpoint at https://huggingface. co/THUMT/mGPT . We propose multi-stage prompting (MSP), a simple and lightweight method for steering pre-trained LMs to translation tasks. We first describe the concept of deep continuous prompts in Section 3.1. Then we detail the stages and training objective in Section 3.2 and Section 3.3, respectively. Finally, we describe the reparameterization of deep continuous prompts in Section 3.4. We adopt continuous prompts (Li and Liang, 2021; Lester et al., 2021) instead of using textual prompts in our method. Using continuous prompts allows learning through differentiable methods like back-propagation (Lester et al., 2021). To be specific, we use deep continuous prompts which are in the same form as in (Li and Liang, 2021). Formally, a prompt P is a sequence of L continuous vectors [ p 1 , . . . , p L ] . Each vector p i (1  i  L ) is a concatenation of key-value pairs in all N Transformer layers, which directly affect the computation of every attention layer. Therefore, the dimension of p i is 2 N  d . We give an illustration of conditioning on a deep continuous prompt in Figure 2. 3.2 Stages To effectively mitigate the semantic and language gap between the pre-training and translation, we 6133 p ( e ) 1 p ( e ) 2 mGPT x 1 x 2 x 3 x 4 x 5 h ( e ) 1 h ( e ) 2 h ( e ) 3 h ( e ) 4 h ( e ) 5 mGPT x 1 x 2 x 3 x 4 x 5 p ( r ) 1 p ( r ) 2 h ( r ) 1 h ( r ) 2 h ( r ) 3 h ( r ) 4 h ( r ) 5 p ( d ) 1 p ( d ) 2 mGPT y 0 y 1 y 2 y 3 y 4 y 1 y 2 y 3 y 4 </S> The Encoding Stage The Re-Encoding Stage The Decoding Stage Figure 3: Detailed computations involved in the multi-stage prompting for machine translation tasks. propose multi-stage prompting which divides the procedure of using pre-trained LMs as translators into three separate stages: the encoding, the re-encoding, and the decoding stages. Given different prompts at different stages, the pre-trained LM is expected to behave differently during each stage and is more capable of generating translations. Given a source sentence x = [ x 1 , . . . , x S ] and a target sentence y = [ y 1 , . . . , y T ] , the details of the three stages are described as follows: The Encoding Stage. At the encoding stage, the pre-trained LM encodes the source sentence x into a sequence of activations H 1: S e by using an encoding stage prompt P e . This procedure is the same as basic prompting. Formally, it can be described as follows: G 1: S e , H 1: S e = f LM ( X 1: S , P e ) . The Re-encoding Stage. At the re-encoding stage, the pre-trained LM produces fine-grained representations of the source sentence by re-encoding x given past activations H 1: S e and a re-encoding stage prompt P r , which allows each representation to condition on all words in x . This procedure can be described as G 1: S r , H 1: S r = f LM ( X 1: S , (cid:74) P r ; H 1: S e (cid:75) ) , (5) where (cid:74) P r ; H 1: S e (cid:75) denotes the concatenation of two sequences P r and H 1: S e . It is also possible to employ more than one re-encoding stage, allowing the pre-trained LM to obtain further refined representations of the source sentence. The Decoding Stage. Finally, we obtain the hidden vectors G 1: T d for predicting the probability of the target sentence y at the decoding stage, given the refined source representations H 1: S r and a decoding stage prompt P d : G 1: T d , H 1: T d = f LM ( Y 1: T , (cid:74) P d ; H 1: S r (cid:75) ) . More precisely, the reparameterization of the three prompts are as follows: P e = max(  e , 1 . 0)   e , (8) P r = max(  r , 1 . 0)   r , (9) P d = max(  d , 1 . 0)   d , (10) 6134 where  e  R 2 N  d ,  r  R 2 N  d , and  d  R 2 N  d . (6) Figure 3 gives a detailed illustration of MSP. By dividing the translation process into multiple stages and applying different prompts, we expect the pre-trained LM model can generate better translations. We use the cross-entropy loss for learning prompts. Given G 1: T d = [ g ( d ) 1 , . . . , g ( d ) T ] in Eq. (6), the training objective is formally described as follows: L =  1 TT (cid:88) t =1 log P ( y t | y <t , x ) =  1 TT (cid:88) t =1 log exp ( e T z t  g ( d ) t ) (cid:80) | V | i =1 exp ( e T z i  g ( d ) t ) . (7) Note that the parameters  of the pre-trained LM are fixed during training. Li and Liang (2021) suggest that using a neural network to reparameterize continuous prompts is more robust to different choices of hyperparameters. In contrast to their approach which uses an MLP network to reparameterize continuous prompts, we introduce a much simpler scaled reparameterization method, in which a continuous prompt is reparam-eterized as a product of a learnable scalar and an embedding. e ,  r , and  d are initialized to 1.0 at the beginning of training. Therefore, the set of trainable parameters  in our method is  = {  e ,  r ,  d ,  e ,  r ,  d } , which contains much less tunable parameters than an MLP network. Scaled reparameterization enables directly adjusting the value of prompts by a tunable scaling factor, leading to a much faster convergence without loss of performance. Further analysis is presented in Section 4.7. Datasets We conduct experiments on Romanian-English (Ro-En), English-German (En-De), and English-Chinese (En-Zh) translation tasks to verify our proposed method. For the Ro-En translation task, we used the WMT16 Romanian-English dataset, which consists of 0.6M bilingual sentence pairs and 2M back-translated sentence pairs. 5 We used newsdev2016 as the development set and new-stest2016 as the test set. For the En-De translation task, we used the WMT14 English-German dataset, which consists of 4.5M sentence pairs. The development set is newstest2013 and the test set is newstest2014 . For the En-Zh translation task, we used the WMT20 English-Chinese dataset as the training corpus, which consists of 28M sentence pairs. The development set is newstest2019 and the test set is newstest2020 . The details of preprocessing and postprocessing are given in Appendix A.2. Metric. We used case-sensitive BLEU (Pap-ineni et al., 2002) as the evaluation metric. The BLEU score is calculated using the SACREBLEU toolkit (Post, 2018). 6 Baselines. We used the mGPT model as the backbone LM in all our experiments, which contains 560M parameters. We compare our method with the following prompting methods: 7  Prompt tuning (Lester et al., 2021). 5 http://data.statmt.org/rsennrich/wmt16_ backtranslations/ro-en 6 Signature: nrefs:1|case:mixed|eff:no|tok:{13a,zh}| smooth:exp|version:2.0.0 7 In our preliminary experiments, we also experimented with the few-shot approach as described in (Brown et al., 2020). However, we found mGPT often failed to generate meaningful translations. Prefix-tuning (Li and Liang, 2021). A prompting method that uses deep continuous prompts, which prepend virtual tokens to all key-value pairs in attention layers of pre-trained LMs. We use an MLP network to reparameterize a continuous prompt during training as suggested in (Li and Liang, 2021). Implementations. All our models are trained on a machine with 8 RTX 3090Ti GPUs. For all prompting methods, we set the prompt length to 128. For the training, we use the Glorot uniform initilalizer (Glorot and Bengio, 2010) to initialize tunable parameters unless otherwise noted. We use Adam (Kingma and Ba, 2015) (  1 = 0.9,  2 = 0.98 and  = 1  10  9 ) as the optimizer with a batch size of roughly 32K tokens. We use the same learning rate schedule as described in (Vaswani et al., 2017). The number of warmup steps is set to 4K. We set the maximum learning rate to 0.02 for prompt tuning and MSP, and 7e-4 for prefix-tuning. 8 We train prompts for a total of 80K steps for prompt tuning and prefix-tuning, and 40K steps for MSP. For the inference, we use the beam search algorithm to obtain translation from the mGPT model, and the beam size is set to 4. The length penalty is determined by the results evaluated on the development set. We set the length penalty to 1.0 for the En-Zh translation task and 0.0 for other translation tasks. We implement our models on top of the THUMT (Tan et al., 2020) toolkit and the Transformers library (Wolf et al., 2020). Table 1 shows the results for the Ro-En, En-De, and En-Zh translation tasks. As the most parameter-efficient among the three prompting methods, prompt tuning introduces only 131K parameters during training for each translation task. However, it only achieves 9.4 BLEU points on average over the three translation tasks. Lester et al. (2021) indicate that language model capacity is a key ingredient for prompt tuning to succeed. As mGPT is a pre-trained LM with only 560M parameters, the results coincide with the conclusion of Lester et al. (2021). Prefix-tuning, which uses deep continuous prompts, achieves an average of 23.9 BLEU points over the three translation tasks. The results indicate that using deep continuous prompts is beneficial 8 We found using a large learning rate for prefix-tuning would result in unstable training. for steering mGPT to translation tasks. However, introducing deep continuous prompts inevitably requires more free parameters. The MLP network used in prefix-tuning introduces about 26M parameters for each translation task during training in our experiments. Finally, MSP achieves 28.0 BLEU points on average over the three translation directions and outperforms prompt tuning and prefix-tuning by 18.6 and 4.1 BLEU points, respectively. MSP introduces 19M parameters for each translation task during training, which is more than prompt tuning but less than prefix-tuning. MSP explicitly divides the translation process using mGPT into separate stages, which are not present in prompt tuning and prefix-tuning. The results suggest that MSP is more effective in instructing pre-trained LMs to perform translation than prompt tuning and prefix-tuning. Table 2 gives the results of mT5-XXL (Zhang et al., 2021), CPM-2 (Zhang et al., 2021), Ernie 3.0 (Sun et al., 2021a), and mGPT on the WMT20 En-Zh translation task. Except for mGPT, other LMs are based on the encoder-decoder architecture. Despite using a much smaller pre-trained LM with about 5% parameters of mT5-XXL, CPM-2, and Ernie 3.0, MSP achieves the best performance on the En-Zh translation task. Therefore, we show that MSP is an efficient and effective approach to steering pre-trained LMs to translation tasks. We compare our method with the state-of-the-art Transformer NMT model (Vaswani et al., 2017) 9 on the TedTalks dataset (Blackwood et al., 2018) and the WMT14 English-German dataset. TedTalks dataset is an English-centric multilingual corpus including 59 languages with around 3K to 200K sentence pairs per language pair. For the sake of simplicity, we only report results for 5 selected languages that contain more than 150K sentence pairs. However, the Transformer model is trained on all available parallel sentences covering 59 languages, serving as a strong NMT baseline. For mGPT with MSP, we individually train the model on each language pair following the same procedure described in this paper. The results of X  En and En  X directions are shown in Table 3. Although mGPT with MSP is independently trained on each language pair, the model still outperforms the strong multilingual NMT baseline by 3.4 and 3.9 BLEU points on X-En and En-X directions, respectively. The results demonstrate that using pre-trained LMs as translators with an appropriate prompting method has the potential to excel a strong Transformer NMT model. Table 4 shows the comparison between Transformer and our mGPT model with MSP on the En-9 We used the transformer-big setting. De translation task. While there is still a noticeable performance gap between Transformer and mGPT with MSP, using mGPT as a translator with MSP is much more parameter-efficient than training a separate NMT model. Supporting En-De translation with mGPT only introduces 19M parameters with MSP method. In comparison, the model size of the Transformer model for En-De translation is 450M. While mGPT model can perform other downstream tasks by providing different prompts, such abilities have not been validated on the Transformer NMT model. Besides being efficient in disk spaces, learning prompts for the En-De translation task are also faster than training a separate NMT model. It takes 21 hours to train prompts for MSP, whereas 72 hours for training a Transformer model. Figure 4 shows the effect of prompt length for prefix-tuning and MSP. We omit the comparison to prompt tuning because of its inferior performance. We found that using longer prompts generally leads to better performance for both prefix-tuning and MSP, but with diminishing returns. This finding is consistent with previous studies (Li and Liang, 2021; Lester et al., 2021). Furthermore, MSP consistently outperforms prefix-tuning when using the same prompt length. Even MSP with a prompt length of 64 performs better than prefix-tuning with a prompt length of 256 (19.0 vs. 18.2). The results further confirm that MSP is a better prompting 64 128 192 256 16 18 20 22 19.0 21.2 22.2 22.4 14.8 17.5 18.2 18.2 Prompt Length BLEU Multi-Stage Prompting Prefix-Tuning Figure 4: Comparison between MSP and prefix-tuning on the WMT14 En-De translation task with different prompt lengths. method than prefix-tuning for steering pre-trained LMs to translation tasks. For the inference time, we found longer prompts do not significantly affect the decoding speed on GPUs as the computation of attention layers are highly parallel, which is also consistent with the findings of Li and Liang (2021). Table 5 shows the comparison of using different stage settings on the WMT14 En-De and the WMT20 En-Zh translation tasks. For single-stage prompting, we also adopt scaled reparameterization instead of MLP reparameterization for a fair comparison. On the WMT14 En-De translation task, using single-stage prompting achieves 17.9 BLEU points. By comparison, explicitly separating encoding and decoding stages improve the translation performance over single-stage prompting by 2.3 BLEU points, which indicates the importance of differentiating stages. Adding a re-encoding stage further improves the translation performance by 1.0 BLEU point, suggesting that the re-encoding stage is effective. Adding a second re-encoding stage further improves the translation performance 6137 Method #Params. by 0.6 BLEU points. Although adding stages introduces more trainable parameters, it should be noted that sharing a single prompt for the encoding/re-encoding/decoding stages also improves over the single-stage prompting by 1.9 BLEU points. The results suggest that most improvements are attributed to the explicit separation of stages rather than increased parameters. Adding more stages generally slows the training speed. However, we do not observe notable inference speed drop as re-encoding stages are computed one time in parallel during inference. On the En-Zh translation task, the results are consistent with the results on the En-De translation task. Therefore, we conclude that using more stages helps improve the translation quality. Figure 5 shows the comparison between MSP using scaled reparameterization and without using reparameterization. Using scaled reparameterization converges faster than without using reparameterization. These two methods achieve nearly the same Prompt Distribution w/o prompt en (16%), ru (10%) Prefix-tuning zh (80%), ja (12%) MSP (encoding stage) en (51%), la (14%) MSP (re-encoding stage) en (24%), la (17%) MSP (decoding stage) zh (91%), ja (9%) Table 6: Language distribution of the free generations using mGPT by conditioning on different prompts learned by different prompting methods on the WMT20 En-Zh dataset. translation performance when the training is converged. As a result, using scaled reparameterization can make the convergence much faster and reduce the total training time. Knowledge. As continuous prompts are learned using bilingual sentence pairs, an interesting question arises: Is the translation knowledge stored in the continuous prompts or the pre-trained LM? To answer this question, we discard the prompts and feed the mGPT model a concatenation of a parallel sentence pair as an input, and calculate the cosine similarities between the source and target hidden activations on each mGPT layer. We found that although the prompts are not given, the nearest pairs of tokens between the source and target language frequently turn out to coincide with bilingual alignments. This finding reveals to some extent that the translation knowledge mainly resides in the pre-trained LM instead of the learned continuous prompts, while the prompts play a role in guiding the model to perform translation during generation. Examples are given in Appendix A.3. Bottleneck. We study the bottleneck of the current prompting method. We train a separate Transformer encoder and an adapter network that directly 6138 maps a source sentence into a deep continuous prompt, leaving the mGPT model only serving as a decoder. This model introduces 378M tunable parameters and achieves 25.9 BLEU points on the WMT14 En-De translation task. Compared with 21.2 BLEU points by MSP, the result shows that there is still room to advance the translation performance of pre-trained LM by improving the prompting method, such as using dynamic prompts (Liu et al., 2021) for each input sentence. However, as translation knowledge may come from the pre-trained LM, the translation performance may be bottlenecked by the capability of the backbone LM. Interpretability. We did not find our learned prompts to be interpretable, which agrees with the findings of Shin et al. (2020) and Lester et al. (2021). However, we do observe prompts of different stages changing the behavior of mGPT significantly. Specifically, we sample 100 examples generated from mGPT by providing prompts of different stages learned on the English-Chinese translation task and identify the language ids of generated texts using the langid toolkit. The top-2 identified language distributions of each generation are shown in Table 6. Without providing prompts, mGPT generates a random sentence from a random language. By given continuous prompts learned by prefix-tuning, the mGPT mostly generates texts related to Chinese. For MSP, it is noticeable that there is a transition from English to Chinese. mGPT generates English-related text given the encoding stage prompt. The distribution of languages becomes smoother when providing the prompt at the re-encoding stage. Finally, mGPT generates Chinese texts dominantly given the decoding stage prompt. The results coincide with our intuition that MSP helps the pre-trained LM to learn smoother transitions to the translation task. Prompting. Brown et al. (2020) propose to use a task description and a few examples to adapt the GPT-3 model to downstream tasks, which is referred to as in-context learning. Their prompts are manually designed. Gao et al. (2020) present LM-BFF for automatic prompts generation. They use T5 model (Raffel et al., 2020) to generate templates for prompting pre-trained LMs. Li and Liang (2021) propose prefix-tuning, which uses continuous vectors as prompts. These prompts are trained using task-specific data and optimized through back-propagation. Lester et al. (2021) propose prompt tuning, which is similar to prefix-tuning but with fewer trainable parameters. Our method is also based on prompting. We use continuous prompts for steering PLMs to translation tasks. Unlike Li and Liang (2021) and Lester et al. (2021) who present general frameworks, our method is focused on improving the translation performance of pre-trained LMs. Using Pre-trained Models as Translators. Stickland et al. (2021) investigate using BART and mBART models for machine translation tasks, their approach relies on adapter networks and finetuning parts of pre-trained LMs. Guo et al. (2020) build a non-autoregressive NMT model by using a source BERT model as the encoder and a target BERT as the decoder with adapter layers. Sun et al. (2021b) propose grafting a source BERT model and a target GPT model for translation tasks. Bapna and Firat (2019) propose using small adapter layers to adapt a base NMT model to new translation tasks. All these methods are adapter-based, which injects additional tunable modules into the pre-trained models. As a result, the pre-trained models lose the ability to perform mixed-task inference. Our approach is based on prompting, which only uses prompts for steering the pre-trained LMs to translation tasks. Zhang et al. (2021) investigate using prompt tuning for steering CPM-2 model to the WMT20 English-Chinese translation task. Furthermore, their approach applied to encoder-decoder architecture pre-trained LMs while ours applied to decoder-only pre-trained LMs. We have presented multi-stage prompting, a method for making pre-trained language models better translators. Experiments show that with multi-stage prompting, pre-trained LMs can generate better translations, showing the potential of using pre-trained LMs for translation tasks. This work was supported by the National Key R&D Program of China (No. 2018YFB1005103), the National Natural Science Foundation of China (No. 62006138, No. 61925601), Institute Guo Qiang at Tsinghua University, and Huawei Noah's Ark Lab. We thank Kehai Chen for the discussion of this work and all anonymous reviewers for their valuable comments and suggestions on this work.
abstain method abstain method method result other abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain objective abstain result other other other other other other other other other other other other other method other method other other other other method other other other method other other abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain method method abstain other abstain abstain abstain abstain abstain abstain abstain method method method method abstain method abstain abstain abstain method abstain abstain method method other method abstain abstain method abstain method method method method method abstain method method method abstain method method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain result other other other other other other other other abstain method abstain other other other other other other other method other other method abstain other other