Abstract Cross-lingual natural language inference (XNLI) is a fundamental task in cross-lingual natural language understanding. Recently this task is commonly addressed by pre-trained cross-lingual language models. Existing methods usually enhance pre-trained language models with additional data, such as annotated parallel corpora. These additional data, however, are rare in practice, especially for low-resource languages. Inspired by recent promising results achieved by prompt-learning, this paper proposes a novel prompt-learning based framework for enhancing XNLI. It reformulates the XNLI problem to a masked language modeling problem by constructing cloze-style questions through cross-lingual templates. To enforce correspondence between different languages, the framework augments a new question for every question using a sampled template in another language and then introduces a consistency loss to make the answer probability distribution obtained from the new question as similar as possible with the corresponding distribution obtained from the original question. Experimental results on two benchmark datasets demonstrate that XNLI models enhanced by our proposed framework significantly outperform original ones under both the full-shot and few-shot cross-lingual transfer settings. Cross-lingual language understanding (XLU) plays a vital role in multilingual systems. It aims at training a model in a source language which is then applied to other languages. Cross-lingual natural language inference (XNLI) is a challenge task for evaluating XLU (Conneau et al., 2018). Natural language inference (NLI) aims to determine the inferential relationship between the text of a premise and the text of a hypothesis while XNLI upgrades NLI to the cross-lingual scenarios. Nowadays pre-trained cross-lingual language models (Conneau and Lample, 2019; Conneau et al., 2020) have become a dominant paradigm for XLU, significantly improving the performance in various XLU tasks included XNLI. Existing methods (Huang et al., 2019; Chi et al., 2021a,b) usually utilize various auxiliary tasks to improve the cross-lingual transferability of a pre-trained cross-lingual language model, mainly relying on annotated parallel corpora. In practice, these methods can hardly work for low-resource language scenarios where parallel corpora are rare. Recently, prompt-learning based methods (Schick and Schtze, 2021; Shin et al., 2020) have shown to achieve promising results for few-shot natural language processing (NLP). These methods reformulate the text classification problem into a masked language modeling problem. In particular, the work (Zhao and Schtze, 2021) demonstrates that prompt-learning outperforms fine-tuning in few-shot XNLI. We argue that the effectiveness of prompt-learning in XNLI still needs to be explored by a larger margin. The reasons are two-fold. On one hand, the effectiveness of prompt-learning in XNLI under the full-shot setting is still unknown. On the other hand, the way to make the best of question templates is unexplored yet. The work (Zhao and Schtze, 2021) uses a uniform template in English for all examples in different languages. This way can hardly capture language-specific characteristics in XNLI, especially for those languages that are right-to-left written such as Arabic and Urdu. We naturally expect that language-specific question templates lead to higher performance in XNLI. Figure 1 illustrates how language-specific question templates are used. The second sub-figure shows the uniform question template used in (Zhao and Schtze, 2021) to handle an Arabic example, where the corresponding example in English is shown in the first sub-figure. The last sub-figure shows the Arabic-specific question template used for the same Arabic example, which is right-to-left written and conforms to the Arabic grammar. In order to introduce language-specific characteristics in question templates while capturing correspondence between different languages, we propose a novel prompt-learning based framework named PCT (shot for Prompt-learning from Cross-lingual Templates ) for XNLI. As illustrated in Figure 2, PCT first constructs a cloze-style question by filling the template in the source language (namely English), then randomly samples a template in another language (such as Chinese) to construct an augmented question, where the augmented question is written in two languages and thus its template is called a cross-lingual template . Both the original question and the augmented question are fed into a pre-trained cross-lingual language model to calculate the answer probability distributions for inferential relationships that are represented by predefined tokens mapped from the mask token. To enforce answer consistency for the two questions, i.e., to make the two probability distributions of inferential relationships as similar as possible, the two probability distributions are regularized by the Kullback-Leibler divergence (KLD) loss. The entire model is trained by minimizing the sum of the cross-entropy loss for classification accuracy and the KLD loss for answer consistency. We employ PCT to enhance pre-trained cross-lingual language models XLM-R (Conneau et al., 2020) and INFOXLM (Chi et al., 2021a). Experimental results on the XNLI (Conneau et al., 2018) benchmark and the PAWS-X (Yang et al., 2019) benchmark show that PCT improves the original models by a significant margin under both the full-shot and few-shot cross-lingual transfer settings. 1. We propose a novel prompt-learning based framework for XNLI. In this framework, a data augmentation strategy is introduced which relies merely on predefined cross-lingual templates; moreover, a consistency loss is introduced to enforce similar output probability distributions for arbitrary two languages so as to capture correspondence between different languages. 2. We conduct extensive experiments on two large-scale benchmarks to demonstrate significant improvements achieved by the proposed framework, under both the full-shot and few-shot cross-lingual transfer settings. Up to date XLU including XNLI are widely addressed by pre-trained cross-lingual language models (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020). Multilingual BERT (mBERT) (Devlin et al., 2019) extends the basic pre-trained language model BERT by training with multilingual corpora. XLM (Conneau and Lam-ple, 2019) enhances mBERT by introducing the translation language modeling (TLM) objective. XLM-RoBERTa (XLM-R) (Conneau et al., 2020) trains XLM with larger corpora and more epochs. Cross-lingual language models can further be enhanced by post-training tasks that rely on large-scale parallel corpora. UNICODER (Huang et al., 2019) introduces several post-training tasks to utilize parallel corpora. INFOXLM (Chi et al., 2021a) enhances XLM-R by introducing the cross-lingual contrastive learning task using 42 GB parallel corpora. XLM-ALIGN (Chi et al., 2021b) introduces a denoising word alignment pre-training task using several parallel corpora. These enhancements can hardly be applied to low-resource languages for which parallel corpora are rare. To alleviate the dependence on parallel corpora, some data augmentation strategies have been proposed for XNLI. TMAN (Qi and Du, 2020) enhances XNLI by exploiting adversarial training from translated data. The work (Dong et al., 2021) proposes a data augmentation strategy for XNLI by generating augmented data from a pre-trained sequence-to-sequence model. UXLA (Bari et al., 2021) improves the performance of XNLI by data augmen-1911 KLD Loss  Yes  Vocabulary <s> I hope to hear from you soon. tation and unsupervised sample selection. All these strategies require a large amount of external resources for data augmentation. In contrast, our proposal augments data by only predefined cross-lingual templates. Recently prompt-learning based methods have shown to achieve promising results in various few-shot NLP tasks. The key of these methods is reformulating the text classification problem into a masked language modeling problem by constructing cloze-style questions. The work (Schick and Schtze, 2021) applies prompt-learning to text classification (including NLI) with manually defined templates. The work (Shin et al., 2020) proposes to search for optimal discrete templates by a gradient based approach. Several approaches (Li and Liang, 2021; Liu et al., 2021; Han et al., 2021) have been proposed to search continuous prompts. The work (Zhao and Schtze, 2021) compares prompt-learning with fine-tuning in few-shot XNLI. Different from (Zhao and Schtze, 2021), this work significantly advances prompt-learning in XNLI further by introducing a new data augmentation strategy and a new consistency loss for regularization. The effectiveness of prompt-learning is also demonstrated further under both the full-shot and few-shot cross-lingual transfer settings. The proposed PCT framework is illustrated in Figure 2. For every training triple (premise, hypothesis, label) in English, PCT first constructs a cloze-style cloze-style question by filling the template in English, then samples a predefined template from another language such as Chinese to construct an augmented question. Both the original question and the augmented question are fed into a pre-trained cross-lingual model to calculate the answer distributions of the mask token, through the masked language modeling (MLM) layer in the pre-trained cross-lingual model. The entire model is trained by minimizing the cross-entropy loss for classification accuracy and the Kullback-Leibler divergence (KLD) loss for answer consistency. The training phase of PCT is formalized in Algorithm 1. For every training triple ( P i , H i , Y i ) in English, where P i = { w Pj } mj =1 denotes the word sequence of the premise, H i = { w H j } n j =1 the word sequence of the hypothesis, Y i  Y the index of the NLI label, PCT first constructs a cloze-style question X i by filling the English template, and then randomly samples a template from other languages to construct an augmented question X i . A template in an arbitrary language is a textual string with three unfilled slots: a input slot [P] to fill the input premise, a input slot [H] to fill the input hypothesis and an answer slot [Z] that allows language models to fill label words. [Z] is usually filled by the mask token [MASK] when using pretrained language models. For instance, the English template is expressed as <s>[P]</s></s>Question: [H]? Answer: [MASK]</s>, where <s> and </s> 1912 Algorithm 1 The training phase of PCT Require: the number of epochs E and the training set D = { ( P i , H i , Y i ) } Mi =1 in English. 1: Reform D to a set of tuples S = { X i , Y i } Mi =1 by filling the English template. 2: Extend S to T = { ( X i , X i , Y i ) } Mi =1 by filling a randomly sampled template from other languages for each ( P i , H i ) . 3: Divide T into a set of mini-batches B . 4: for epoch from 1 to E do 5: Shuffle B . 6: for each mini-batch { ( X i , X i , Y i ) } 1  i  N in B do 7: Compute total loss L by Eq. (5). 8: Update parameters  by gradient descent. 9: end for 10: end for are special tokens in XLM-R to separate sentences. The verbalizer M : Y  V is a function to map NLI labels to indices of answer words in the given vocabulary. Let l denote the size of the given vocabulary and d the dimension of the contextualized representation of a token, output by a pre-trained cross-lingual language model with an MLM layer, such as XLM-R (Conneau et al., 2020). The answer probability distribution is calculated by: y i = softmax( W lm h [MASK] i ) (1) where W lm  R l  d denotes the parameters of the pre-trained MLM layer and h [MASK] i  R d denotes the contextualized representation of the [MASK] token of the i th training triple. Compared with the standard fine-tuning method, no extra parameters are required to be initialized, therefore the model can be optimized by fewer samples. Given a mini-batch ( X i , X i , Y i ) 1  i  N of N triples, the two cross-entropy losses for the original question and the augmented question are respectively calculated by: LX =  1 NN (cid:88) i =1 l (cid:88) j =1 I ( j = M ( Y i )) log y X i i,j (2) LX =  1 NN (cid:88) i =1 l (cid:88) j =1 I ( j = M ( Y i )) log y X i i,j (3) where y X i i,j (resp. y X i i,j ) denotes the j th element of y i  R l for the input X i (resp. for the input X i ). More generally, all cross-lingual templates are obtained from the English template by translating prompt words in English to prompt words in other languages using Google translator 1 , where for Arabic 1 https://translate.google.com/ 1913 and Urdu, the prompt part is written from right to left rather than from left to right as in English and other languages. We observe that, given the same input premise and hypothesis, the answer probability distribution of the question constructed by a cross-lingual template may evidently deviate from that of the question constructed from the English template. Such a deviation may lead to an increase of errors when applying cross-lingual templates to examples in other languages. Our ablation study in Section 4 confirms this phenomenon. To eliminate the negative effect of this deviation, we propose a consistency loss function to regularize the answer probability distributions. More precisely, we employ the symmetric Kullback-Leibler divergence (KLD) loss to enforce the answer probability distributions y X i i and y X i i to be as similar as possible, which is formally defined bellow. LKLD = 1 NN (cid:88) i =1 (KL( y X i i || y X i i ) + KL( y X i i || y X i i )) = 1 NN (cid:88) i =1 l (cid:88) j =1 ( y X i i,j log y X i i,j y X i i,j + y X i i,j log y X i i,j y X i i,j ) (4) The entire model is trained by minimizing the total loss L formally defined as: L = LX + LX + LKLD (5) where we simply apply the same weight for the three loss terms. Since the English template may not conform to the grammar of other languages such as Arabic and Urdu, PCT uses the cross-lingual template in the target language for predicting test examples in the target language. For instance, every Chinese test example is reformed to a Chinese cloze-style question by filling the Chinese template <s>[P]</s></s>   : [H]?   : [MASK]</s>, which is obtained from the English template by translating prompt words in English to prompt words in Chinese, where the slots [P] and [H] are filled by the premise and the hypothesis in Chinese, respectively. By considering that the English label words have been fine-tuned to work for different languages during the training phase, we use the same English verbalizer M for all languages in the inference phase. To evaluate the effectiveness of the proposed PCT framework, we applied PCT to enhance several pretrained cross-lingual language models including XLM-R base , XLM-R large and INFOXLM large . We call the enhanced models PCT-X, where X denotes the original pre-trained cross-lingual model. XNLI : The XNLI (Conneau et al., 2018) benchmark 2 extends the MultiNLI (Williams et al., 2018) benchmark (in English) to 15 languages through translation and comes with manually annotated development set and test set. For each language, the training set comprises 393K annotated sentence pairs, whereas the development set and the test set comprises 2.5 K and 5K annotated sentence pairs, respectively. PAWS-X : The PAWS-X (Yang et al., 2019) is a cross-lingual paraphrase identification benchmark 3 , which extends the Wikipedia portion of the PAWS (Zhang et al., 2019) dataset to 7 languages through translation. For each language, the training set comprises 49.5K annotated sentence pairs, whereas both the development set and the test set comprise 2K annotated sentence pairs each. We implemented our enhanced models by Tensor-flow 2.4.0 and trained all the models with 8 TPUs on the Google Colab platform 4 . PCT-XLM-R base was initialized by the pretrained XLM-R base model with 12 transformer layers, which outputs 768-dimensional token embed-dings. The transformer encoder was built with 12 heads. We applied dropout (Srivastava et al., 2014) to each layer by setting the dropout rate to 2 http://www.nyu.edu/projects/bowman/ xnli/ 3 https://github.com/ google-research-datasets/paws 4 https://colab.research.google.com/ 0.1. The model was trained by Adam (Kingma and Ba, 2015) with the warmup mechanism (Devlin et al., 2019) and two training epochs, where the initial learning rate was set to 5e-5, the warmup proportion to 10%, and the mini-batch size to 64. PCT-XLM large and PCT-INFOXLM large were respectively initialized by the pre-trained XLM-R large and INFOXLM large models with 24 transformer layers, both of which output 1024-dimensional token embeddings. The transformer encoder was built with 16 heads. The models were trained by RMSProp (Dauphin et al., 2015) with one training epoch, where the initial learning rate was set to 5e-6, the mini-batch size to 32, and the dropout rate to 0.1. We used RMSProp instead of Adam for these large models since the training memory is limited by the Google Colab platform. For all the above models, the input sentence pairs were truncated to maximum 128 tokens. Code and data about our implementations are available at https://github.com/qikunxun/PCT . We compared our models with the following pretrained cross-lingual language models: (1) multilingual BERT (mBERT; Devlin et al. (2019)) is a BERT model pre-trained on Wikipedia with 102 languages; (2) XLM (Conneau and Lample, 2019) is pre-trained for two tasks (MLM and TLM) on Wikipedia with 100 languages; (3) XLM-R (Con-neau et al., 2020) extends XLM with larger corpora (i.e. the CC-100 corpora with 100 languages) and more training epochs; (4) UNICODER (Huang et al., 2019) continues training XLM by introducing several post-training tasks using parallel corpora; (5) INFOXLM (Chi et al., 2021a) enhances XLM-R by introducing the cross-lingual contrastive learning task using 42 GB parallel corpora; (6) XLM-ALIGN (Chi et al., 2021b) enhances XLM-R by introducing the denoising word alignment pre-training task using several parallel corpora; (7) The work (Dong et al., 2021) proposes an adversarial data augmentation strategy for XNLI based-on XLM-R; (8) UXLA (Bari et al., 2021) extends XLM-R with data augmentation and unsupervised sample selection. (9) The work (Zhao and Schtze, 2021) proposes three prompt-learning methods for few-shot XNLI, including DP (direct prompting), SP (soft prompting) and MP (mixed prompting). We conducted experiments on both XNLI and PAWS-X under the cross-lingual transfer setting, where models are trained on data in the source language (usually English) and tested on data in the target language. This setting is commonly used to evaluate XNLI models. It can be further divided into two sub-settings: the full-shot setting using the whole training set, and the few-shot setting using a fixed number of training samples. For both XNLI and PAWS-X we evaluated models under the full-shot setting, whereas for XNLI we additionally evaluated models under the few-shot setting. Table 1 reports the results for comparing PCT-enhanced models with other models on XNLI under the full-shot setting. The results of compared models are taken from (Chi et al., 2021a) and (Liang et al., 2020). PCT-XLM-R base achieves 75.3% accuracy on the XNLI test set averaged by 15 target languages, significantly outperforming its basic model XLM-R base by an absolute gain of 1.1% accuracy on average. The difference between PCT-XLM-R base and XLM-R base in average accuracy is statistically significant with p-value 1.7e-6 by a two-tailed t-test. Meanwhile, PCT-XLM-R base outperforms the three prompt-learning approaches (i.e. DP-XLM-R  base , SP-XLM-R  base and MP-XLM-R  base ) in (Zhao and Schtze, 2021) under the full-shot setting. PCT-XLM-R large achieves 81.3% accuracy on the XNLI test set averaged by 15 target languages, pushing XLM-R large by an absolute gain of 0.9% accuracy on average. The difference between PCT-XLM-R large and XLM-R large in average accuracy is statistically significant with p-value 2.5e-4 by a two-tailed t-test. Furthermore, it can be seen that the average accuracy of PCT-XLM-R large is close to that of the current state-of-the-art model INFOXLM large (i.e. 81.4%), which is trained with additional data. To further verify the effectiveness of PCT, we also applied PCT to INFOXLM large , denoted by PCT-INFOXLM large . It can be seen that PCT-INFOXLM large achieves 82.0% accuracy on average, pushing INFOXLM large by an absolute gain of 0.6% on average. The difference between PCT-INFOXLM large and INFOXLM large in average accuracy is statistically significant with p-value 7.5e-3 by a two-tailed t-test. These results imply that PCT is able to further improve the cross-lingual transferability of state-of-the-art models. Table 2 reports the comparison results on PAWS-X under the full-shot setting. The results of compared models are taken from (Hu et al., 2020). Since the work (Hu et al., 2020) has not reported the result of XLM-R base , we produced the result of 1915 Shots Models en fr es de el bg ru tr ar vi th zh hi sw ur  K=16 FT 34.7 33.8 33.8 34.3 33.5 33.8 34.1 34.1 33.6 34.0 33.1 33.5 33.1 33.7 33.2 33.7 DP 38.2 36.6 36.9 37.5 37.4 37.1 36.5 35.7 35.1 35.8 37.2 37.9 35.9 33.8 34.9 36.4 SP 39.5 40.9 39.4 40.2 40.4 40.6 40.6 36.3 38.9 38.5 39.5 37.4 36.9 37.1 35.9 38.8 MP 33.2 34.4 34.5 34.0 32.6 33.0 33.9 34.7 32.5 33.3 33.5 35.7 34.3 33.3 32.7 33.7 PCT (this work) 46.5 44.3 41.5 36.9 45.7 40.8 42.4 43.7 43.6 44.7 43.9 44.8 44.8 40.1 42.5 43.1 K=32 FT 36.6 36.5 36.0 36.0 36.1 36.3 35.7 35.9 35.8 36.1 35.7 35.7 36.2 35.3 34.8 35.9 DP 43.7 43.9 42.8 43.5 42.5 43.5 42.5 42.0 41.8 41.9 40.5 39.9 39.3 37.5 39.8 41.7 SP 44.7 42.3 42.3 42.1 42.3 43.4 43.8 38.8 40.3 42.1 40.0 39.6 38.9 37.5 38.8 41.1 MP 45.5 44.7 41.2 42.6 42.3 42.2 42.2 41.2 41.0 41.7 40.2 40.9 40.2 36.5 40.5 41.5 PCT (this work) 49.6 48.8 45.5 44.4 47.4 45.4 45.5 44.3 45.7 46.7 41.6 45.6 46.7 40.3 42.9 45.4 K=64 FT 41.7 39.5 40.3 40.1 39.9 39.6 38.3 39.5 40.2 40.9 39.2 39.6 39.5 39.6 39.2 39.8 DP 48.9 48.0 45.0 48.1 46.9 47.6 44.9 45.7 45.6 47.3 45.7 45.2 41.6 41.0 43.3 45.7 SP 49.0 46.1 45.8 46.0 43.7 43.8 44.5 41.9 43.5 45.3 44.7 44.2 40.9 40.5 40.1 44.0 MP 51.8 48.3 46.6 48.2 46.8 46.0 44.8 44.8 43.9 48.3 45.0 43.0 40.1 37.8 44.0 45.3 PCT (this work) 51.5 51.3 50.9 49.3 50.6 50.2 49.1 47.4 48.1 49.7 47.3 48.2 47.6 44.6 44.0 48.6 K=128 FT 46.9 46.0 45.8 45.6 44.4 45.5 44.9 43.7 43.5 44.8 43.3 44.8 43.0 41.4 41.8 44.4 DP 53.7 49.3 48.5 51.0 47.4 50.5 46.9 49.6 46.2 48.9 44.8 49.6 44.8 42.0 44.2 48.0 SP 49.5 46.4 45.8 45.0 46.3 46.2 45.0 41.9 44.8 45.0 45.6 45.7 43.3 41.2 41.2 44.9 MP 52.6 50.3 49.7 49.0 49.1 48.0 46.4 48.5 46.5 48.2 48.1 50.5 47.0 42.9 44.0 48.0 PCT (this work) 55.0 53.3 53.8 52.8 53.4 51.9 51.7 50.9 50.4 51.7 50.0 51.2 51.5 47.0 47.9 51.5 K=256 FT 57.8 55.4 55.9 54.4 54.0 54.6 52.9 52.3 52.1 54.2 51.2 52.1 50.7 50.0 48.6 53.1 DP 60.1 54.4 50.6 55.4 55.1 55.6 51.4 50.8 53.2 55.1 53.4 52.7 46.1 45.3 48.4 52.5 SP 60.6 55.8 54.8 53.0 53.1 56.0 52.5 52.1 52.3 54.5 54.5 54.6 49.4 47.3 48.5 53.3 MP 60.1 55.3 51.6 50.7 54.6 54.0 53.5 51.3 52.8 52.3 53.4 53.8 49.6 45.3 47.2 52.4 PCT (this work) 60.3 58.3 58.3 56.3 57.9 56.7 55.2 54.6 54.7 57.4 55.6 55.8 54.6 51.6 52.6 56.0 Table 3: Comparison results on XNLI under the few-shot setting. XLM-R base (denoted by XLM-R  base ). PCT-XLM-R base achieves 85.4% accuracy on the test set averaged by 7 languages, pushing XLM-R base  by an absolute gain of 1.1% accuracy on average. The difference between PCT-XLM-R base and XLM-R base  in average accuracy is statistically significant with p-value 3.2e-3 by a two-tailed t-test. PCT-XLM-R large achieves 88.3% average accuracy on the PAWS-X test set, pushing XLM-R large by an absolute gain of 1.9% accuracy on average. The difference between PCT-XLM-R large and XLM-R large in average accuracy is statistically significant with p-value 3.2e-3 by a two-tailed t-test. Table 3 reports the results for comparing PCT-XLM-R base with all approaches proposed in (Zhao and Schtze, 2021). Note that all compared models are based on XLM-R base and we evaluated PCT-XLM-R base using the same split of data from (Zhao and Schtze, 2021). The training and validation data are randomly sampled by (Zhao and Schtze, 2021) with K  { 16 , 32 , 64 , 128 , 256 } shots per class from the English training data in XNLI. Results show that PCT-XLM-R base statistically outperforms all baselines in all experiments. In particular, PCT-XLM-R base outperforms the fine-tuning baseline by an absolute gain of 9.4% accuracy on average in the 16-shot experiments. It can also be seen that the difference between PCT-XLM-R base and fine-tuning baseline becomes larger as K decreases, implying that the PCT framework becomes more effective when training data are fewer. We also evaluated PCT on XNLI under the TRANSLATE-TRAIN-ALL setting, where all translated data are used in training, to see how well PCT is adapted to this setting. We construct an original question from the template of each of the 15 languages and an augmented question from a sampled template of other languages. Table 4 reports the comparison results. PCT-XLM-R large under this setting achieves significantly better performance than under the cross-lingual transfer setting, but fails to outperform its original model XLM-R large . This inferiority may be caused by the relatively low quality of examples in source languages. Note that an example in a source language other than English is translated from an English example 1916 Variant Models en fr es de el bg ru tr ar vi th zh hi sw ur  p-value Original PCT-XLM-R base 84.9 79.4 79.7 77.7 76.6 78.9 76.9 74.3 72.9 76.0 72.0 74.9 71.7 65.9 67.3 75.3 (1) W/o the consistency loss 84.6 79.6 79.5 76.7 76.3 78.1 76.0 73.9 72.1 75.0 72.3 73.9 71.1 63.9 66.8 74.7 1.2e-3 (2) W/o the PCT framework 83.9 78.1 78.5 76.1 75.7 77.1 75.3 73.2 71.6 74.7 70.9 73.4 70.2 63.6 65.5 73.9 1.5e-9 (3) Using cross-lingual templates in (2) 83.9 77.4 78.3 75.6 75.1 76.5 74.8 72.3 70.8 74.3 70.6 72.0 69.7 63.7 65.0 73.3 3.0e-10 (4) W/o the cross-lingual templates 84.8 79.5 79.6 77.9 76.4 78.2 76.7 74.2 72.5 76.0 71.9 74.6 71.6 64.8 66.9 75.0 3.0e-2 (5) Using substitute word templates 84.6 79.0 79.6 77.1 76.5 77.9 75.9 73.8 72.0 75.5 71.5 73.9 70.6 66.3 65.8 74.7 4.2e-4 Table 5: Ablation study results for PCT-XLM-R base . and may have translation errors. As a future work, we will go on studying whether using training data in multiple languages helps to improve XNLI by collecting more real-world data in other languages. Table 5 reports the ablation study results for PCT-XLM-R base . For the variant (1), we omit the consistency loss in course of training. Results show that the usage of consistency loss achieves better performance on average. For (2), we omit the whole PCT framework in course of training. Results show that the usage of PCT pushes XLM-R base with standard prompt-learning by an absolute gain of 1.4%. For (3), we apply the cross-lingual templates to the variant (2). Results show that the performance drops about 0.6% on average when applying only the cross-lingual templates. For (4), we use only the English template in the inference phase. Results show that PCT-XLM-R base achieves better performance on average when the cross-lingual templates are used in inference. For (5), we use the substitute word templates for Arabic and Urdu as for other languages, i.e., the templates for Arabic and Urdu are also left-to-right written. Results show that PCT-XLM-R base is able to capture certain language-specific characteristics in the target language to achieve better performance. To clarify why the proposed PCT framework improves accuracy in predicting NLI labels, we visually compared the representations of the [MASK] token generated by standard prompt-learning based XLM-R base (denoted by PL-XLM-R base ) with that generated by PCT-XLM-R base , by using t-SNE (Laurens and Hinton, 2008) to reduce the dimension. The results are shown in Figure 3. For the sub-figures (a) and (d), the points marked with x", +" and o correspond to examples with the label entailment, contradiction and neutral, respectively. The points with different colors correspond to examples in different languages. The figures were obtained by randomly selecting 200 examples for each language from the XNLI test set. It can be seen in (a) that a group of red points (for Urdu) and purple points (for Arabic) are dissociated while all points from different languages are mutually overlapped in (d). Considering that the the points from Arabic and Urdu are quite different, we further analyzed them. For the sub-figures (b), (c), (e) and (f), the points marked with o" and +" respectively correspond to examples in English and in either Arabic or Urdu. The points with blue, red and green color correspond to examples with the label entailment, neutral and contradiction, respectively. Sub-figures (b) and (e) (resp. sub-figures (c) and (f)) were obtained by randomly selecting 1000 examples in English and 1000 in Arabic (resp. in Urdu) from the XNLI test set. Compared with PL-XLM-R base , PCT-XLM-R base yields clearer distinction between different labels and more confusion between English and the target language (Arabic or Urdu). These results imply that the PCT framework tends to align contextualized representations in different languages into the 1917 ++o + x + x oo ++ o + xo +xx o o o o + + + x o+ x + x x x o + x xx + x x + o x + x + + x x o x o o o + + x o + xx o + o o oo x + o + o x x x o o + + + x+ x x + + + o x + x o x + o + o + + o oo x x o+ x x + xo x o + + x+ + +++ o + o o + o o o+ x o x o+o x x o + o xx + o x x + x o o + o + x o + + x x o o x + x o x o x + x x + x +o o x x + x o o o o xx ++ + o x o x + x + o x o o o + o + o x x o o o + o o ox o + + x + x o o +x o o o o o + x + x o x o o + x o o x + o x x + o o + + x o o+ o o + + x o x + + x o o + x o o x o + + x x o o x o x x o x + x + + + o x x x +o o x o o o + + + + + o + o x o + x ++ + + x o x x o o o + x x o + o o o x o o x + o + o x + + + x + + o xx + + x x o + o o x x o x + x o x x o x + + x x o o x o x x + + o x oo + x + o o + + + o o o o + x x x o + x ++ox+ o o x o o o + x oo x+ o o + o o x + o + o +x + + x o o + x + x o o x + o + +x + + + x x o x o x + x o + o x x o x +o + o o + o + ++ x o + x x o + + o ++ + x x o o + + + o o x o x+ x + o x + xo + o o o + o x x o x oo x x + o + x + + +o x x + o o + o + o x + ox o o + x x o x + x x o o + + x + o x o + x o x x x x x x o + x + + o o o x xx o + o o o o + + + o + + + + o + x + o + x o + x oo x + o + + x o x + o o o o o o x o o o o o x x x x + x + + x o o x o + o ++ x x + o + o + x x o o o + + + x x x + x x o + + x x o + + x o o x x o o + o x o + o x + o x o o x o x + + x + x + + x + x x + x + o x xx o + o x o o x o o o + + + + o o x o o + x + o o o x x o x x + x o x x x + x + o o o o + x o x + + x x x o x o oo x + x o + x x o x x o o + o x x x x + + o x + x o x + x + x + x + x + o o o + + + + x x o o x x o x x + + o o o x x x o x o o x o o o o + x o o o + + x + o o x x + x + o + + x x o o + + o x + x + o + x o x o x o + x x x x o + + + o oo + x o + o + x x x x x + + + + + x oo x o x + o + o o x o o x o + o x o x x o o x x + x x + o x o x + o xo o + x x o o o + + xx o x o + + o o o x x o + x + o x o + x o o x x x o o x o + x o + o ++ x o x + x + x ooo + o x x x + x + o + x + + x + + x x x x + + x o + x x x x o o x x x o x x o + o + x o o o + x + o + x o o + + + x o o + x o + o x o x o + + x + o o x x o + + + + + x x o x o x o x + o x o + + x o + + + x x + + o x o x + + x o + + + o x + x x x x o x + + + x x o x + o + o o + x + x o o + o x o x + x o + + o + o o x + x x x x oo + x o o o + o o x o o x xx + o o x o o o x + o + o o + + + + + o o x + o + o ++x o+x o x x x o x x x + x + x x + x o o o o x o x o o o ++ x x + o + + x x + x o x x o o + xx + xo o o + + + o o o o x o + x + + o + x + o x o x ++ o + + xx + x x o o o + x ++ + x + + + + o o o o x + o x o x + o x x oo+ xx x x + + oxo + + + + x o o x o x x+ o x x xo x + x + ++ x + o + + + + o ox x + x x x x ++ + + o x o + xxx + x+o+ x x x x x o o + o x ox + + o + o + x + x xx +o o + x o x x x x o o x + + + x x o x o + o x o o x o x o x o + o x + + + x + o x x o x + + o x x xo + x x x + o o o o o o x x o + o o + + + o x o + x o x + x + x x + x x x + o x x o x x x o + o x x x x +o o x o o o + o + + o o + o o o x x x + x o x x x x x o x + o + +x + + o o + x o x x x x o o + o x x o o + x o + o x x o + o o + o + o x + x + o x x + o + x x o x + o + o x o + x o + x o o o x x + x + o x + x x o + o o + + + o x o o + o + x o + + + + x x o x + + x o x + x o x x x o + + o o o + ++ x o x + o + + o o o + o o + + x o x o x o x x x + xx o + x xx x x o + x + o + x o x o + + +o + x x + + x x + o x o + x o + o x + o x x x x o o o x + + o + o + o o o + x x o x o x + x x x x + o o x x o o o o x x + + o x + + + x x + x + x + o + x o o x x x x o o x o o o o + x + x + x o x x + + x o o + x + x + + x x o + x x + x o + x + + o o o x + + + x x o o x o x + o x + + + o + x + x x x + o x x o x + o xo + x + + x + + + x + + o o + + + x o x x x + + o o o x o + + x x o x x o x x + + + o + + x x + x o x x x xx o x + x + o o o o x + x o o x + x o x x + x x x + + o o x + + + o o x o x o + + + o x o x x o + x x + x + x o x x o o + x + o + x + x oo + x o o + o + + + o + + x o o o + x x + + o o + o x x + x o o x oo x x + + o o o o x x x x x x o + o o + x o + x o o o o x + + x + o + o x x o o x o + o o + x + o + + +o o o x + o x o + + x x + x + + o o + xx xx o x x + o + x o o o x o x + x o + o + + + o o x + o o + o xx + o + + + o + + o + x o o + o x x + x o x o o + x x x o x x x o x o + + o o o o o x o + + + o o o + + + +o + o + o + o x + x + o o x + + o o o o o o o + o x + x x x x x + o x x x + x + + + x + + + o + x + x + + o o o x + o o o o o + x o x + x + x + x + o + x o + + x x o + + + + o + x x + o + + o + + x o x o + + x o x + x o x o o x x + o o + + + x o + o + o + o o o x + + x x o x o x x o o o x x x + x + + o x o x + + + x x x + x + o + x o o o + o + o x x x o o x + o x + x + x x o + x o x x + o + + + x x o o + x x + + x x x + + o o + + + o + x o + + o x x + + o x x o x x x + x o o o x x x x x x o o + o o + + o x x + o + + + + o x + x + x o x + + x x x o oo o x x o + o + + o + o + + + + + + x + + o + x o + o+ + + o x o x + o xxx+ + + o o x o + + o x x o x x o x o o + x o x + o + + o o x o + x + o x + x o + x x x o + o o x x o + o + o + o o x o + + + + x + x + o o + + o o + o x + o o + x + + o o + o o + x o x o o + + + o x + + o x x o x + x x x + + + o x + o o o o o + + x x x x o + x x x x x x + o x + x + o + + + o o x + x x o o o x x x + x x o + x + o o + x o + + + x o o x + x + x x x o x o + + o + + x x + + o x o o o + x x o x o o + o x + + x o o x x o + x o x o + x x o o o + x + + o o + + + x o + o o + o + x + o + + o o o + o x + o x x + o o o o o + + x + o x o o + + x x + x o + x x x + + + o o + x o x o x + x x + o x + + + o o o o + x o + x + x x x o + + + x x + o x o o x x + x + + + x o x + x + + o + + x oo + x + o x x x + x o + + x o x + + x o x o x x o x + o x o x + x + x o o o + o + + x + + x x o o + + x x o x + o x + + o + o o + o o o + o + o x + o + o + x o + x x x o + x x + + x x x x o + + + x + x ++ x + + o + x x x + + + o x + o + o x + o + x + o x + x + o + x + o x ++ + x xx o + o same space, which helps to improve the prediction accuracy in the XNLI task. We also conducted experiments to show how different strategies for template selection impact the performance. The results are reported in Table 6. We compared the default uniform strategy with two different selection strategies, where one sets the probabilities for selecting XX directly proportional to and the other inversely proportional to the XX-En BLEU scores, which are directly taken from Table 3 in (Conneau et al., 2018) and can be considered as similarity degrees between the target languages XX and English. Results show that the performances of both PCT-XLM-R base and PCT-XLM-R large slightly drop when using the directly proportional strategy. It can also be seen that, PCT-XLM-R base with the inversely proportional" strategy achieves the same average accuracy as with the uniform strategy, while PCT-XLM-R large with the inversely proportional" strategy is lightly better than with the uniform strategy. This implies that the inversely proportional" strategy is able to improve the performance by selecting more templates in target languages that are less similar to English. However, the improvements are not significant as p-value > 0.05 by two-tailed t-tests. By considering that XX-En BLEU scores are not available in most practical scenarios, we recommend to use the uniform strategy for template selection. 5 Conclusions In this paper we have proposed a prompt-learning based framework named PCT for cross-lingual natural language inference. PCT enhances pre-trained cross-lingual language models by augmenting data from cross-lingual templates and by introducing the consistency loss to regularize the answer probability distributions. Experimental results on large-scale benchmarks XNLI and PAWS-X show that PCT pushes existing models by a significant absolute gain in accuracy under both the full-shot and few-shot cross-lingual transfer settings. Our ablation study and visualization analysis further confirm the contributions of different enhancements introduced by PCT. Future work will study PCT further under the TRANSLATE-TRAIN-ALL setting with real-world data in different languages. Acknowledgements This paper was supported by the National Natural Science Foundation of China (No. 61976232 and 61876204), Guangdong Basic and Applied Basic Research Foundation (No.2022A1515011355 and 2020A1515010642), Guizhou Science Support Project (No. 2022-259), Humanities and Social Science Research Project of Ministry of Education (18YJCZH006). 1918 References M. Saiful Bari, Tasnim Mohiuddin, and Shafiq R. Joty. 2021. UXLA: A robust unsupervised data augmentation framework for zero-resource cross-lingual NLP. In ACL , pages 19781992. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS , pages 18771901. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021a. Infoxlm: An information-theoretic framework for cross-lingual language model pre-training. In NAACL-HLT , pages 35763588. Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, and Furu Wei. 2021b. Improving pretrained cross-lingual language models via self-labeled word alignment. In ACL , pages 3418 3430. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmn, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL , pages 84408451. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In NeurIPS , pages 70577067. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: evaluating cross-lingual sentence representations. In EMNLP , pages 24752485. Yann N. Dauphin, Harm de Vries, and Yoshua Bengio. 2015. Equilibrated adaptive learning rates for non-convex optimization. In NIPS , pages 15041512. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , pages 41714186. Xin Dong, Yaxin Zhu, Zuohui Fu, Dongkuan Xu, and Gerard de Melo. 2021. Data augmentation with adversarial training for cross-lingual NLI. In ACL , pages 51585167. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. PTR: prompt tuning with rules for text classification. CoRR , abs/2105.11259. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In ICML , pages 44114421. Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. In EMNLP , pages 24852494. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR . Van der Maaten Laurens and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research , 9(11). Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL , pages 45824597. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark datasetfor cross-lingual pre-training, understanding and generation. In EMNLP , pages 6008 6018. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. CoRR , abs/2103.10385. Kunxun Qi and Jianfeng Du. 2020. Translation-based matching adversarial network for cross-lingual natural language inference. In AAAI , pages 86328639. Timo Schick and Hinrich Schtze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL , pages 255269. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP , pages 42224235. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR) , 15(1):19291958. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT , pages 11121122. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In EMNLP , pages 36853690. 1919 Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: paraphrase adversaries from word scrambling. In NAACL-HLT , pages 12981308. Mengjie Zhao and Hinrich Schtze. 2021. Discrete and soft prompting for multilingual models. In EMNLP , pages 85478555. A Cross-lingual Templates Here we introduce the cross-lingual templates that we used in our experiments. We used the same English template defined by (Zhao and Schtze, 2021) for XNLI and used the English template defined by (Brown et al., 2020) for PAWS-X. The cross-lingual templates are generated by translating the English template to target languages using Google translator. The cross-lingual templates for XNLI are given in Figure 4. The cross-lingual templates for PAWS-X are given in Figure 5. The slots [P] and [H] are filled by the premise and the hypothesis, respectively. B Results with Standard Deviations Here we report the complete experimental results taken from five runs with standard deviations. The means and the standard deviations are reported in the row avg. and s.d., respectively. For XNLI under the full-shot cross-lingual transfer setting, the experimental results are reported in Table 7, including the results for all five runs achieved by PCT-XLM-R base , PCT-XLM-R large and INFOXLM-R base . For PAWS-X under the full-shot cross-lingual transfer setting, the experimental results are reported in Table 8, including the results for all five runs achieved by PCT-XLM-R base , PCT-XLM-R large and INFOXLM-R base . For XNLI under the few-shot cross-lingual transfer setting, the experimental results with K  { 16 , 32 , 64 , 128 , 256 } shots per class are reported in Table 9, including the results for all five runs achieved by PCT-XLM-R base . 1920 Template Language <s>[P]</s></s>Question: [H]? Answer: <mask></s> English (en) <s>[P]</s></s>Question: [H]? Rponse: <mask></s> French (fr) <s>[P]</s></s>Pregunta: [H]? Respuesta: <mask></s> Spanish (es) <s>[P]</s></s>Frage: [H]? Antwort: <mask></s> German (de) <s>[P]</s></s>: [H]? : <mask></s> Greek (el) <s>[P]</s></s>: [H]? : <mask></s> Bulgarian (bg) <s>[P]</s></s>: [H]? : <mask></s> Russian (ru) <s>[P]</s></s><mask> : : [H] : </s> Arabic (ar) <s>[P]</s></s>Soru: [H]? Cevap: <mask></s> Turkish (tr) <s>[P]</s></s>Cu hi: [H]? Tr li: <mask></s> Vietnamese (vi) <s>[P]</s></s>  : [H]?  : <mask></s> Thai (th) <s>[P]</s></s>  : [H]?  : <mask></s> Chinese (zh) <s>[P]</s></s> !" : [H]?
abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain method abstain objective objective objective other other other other other other other other other other other other other other other objective other other other other other other abstain other abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain method abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain other