Cross-lingual retrieval aims to retrieve relevant text across languages. Current methods typically achieve cross-lingual retrieval by learning language-agnostic text representations in word or sentence level. However, how to learn phrase representations for cross-lingual phrase retrieval is still an open problem. In this paper, we propose XPR , a cross-lingual phrase retriever that extracts phrase representations from unlabeled example sentences. Moreover, we create a large-scale cross-lingual phrase retrieval dataset, which contains 65K bilingual phrase pairs and 4.2M example sentences in 8 English-centric language pairs. Experimental results show that XPR outperforms state-of-the-art baselines which utilize word-level or sentence-level representations. XPR also shows impressive zero-shot transferability that enables the model to perform retrieval in an unseen language pair during training. Our dataset, code, and trained models are publicly available at github.com/cwszz/XPR/ . Phrase retrieval aims to retrieve relevant phrases from a large phrase set, which is a critical part of information retrieval. Recent studies on phrase retrieval learn dense representations of phrases, and achieve promising results in entity linking, slot filling, and open-domain question answering tasks (Gillick et al., 2019; Lee et al., 2021a,b). Nonetheless, most of the studies focus on monolingual scenarios, leaving the cross-lingual phrase retrieval unexplored. Various methods have been proposed to perform cross-lingual text retrieval, which learns cross-lingual word or sentence representations shared  Co-first authors with equal contributions. across languages. Cross-lingual word representation methods typically train word embeddings on each language separately, and then learn an embedding mapping between the embedding spaces of different languages (Mikolov et al., 2013; Dinu et al., 2014). Then, the bilingual word pairs can be retrieved between vocabularies using nearest neighbor search, which is also known as bilingual lexicon induction (Artetxe et al., 2018; Lample et al., 2018). Cross-lingual sentence retrieval is typically achieved by learning a sentence encoder on multilingual text corpora with self-supervised pretraining tasks (Conneau and Lample, 2019; Conneau et al., 2020), or large-scale parallel corpora (Artetxe and Schwenk, 2019), or both (Chi et al., 2021b). The trained sentence encoders produce language-agnostic sentence representations, which enables sentences to be retrieved across languages. Despite the effectiveness of word-level and sentence-level methods, how to learn phrase representations for cross-lingual phrase retrieval is still an open problem. Learning cross-lingual phrase representations is challenging in two aspects. First, a phrase is a conceptual unit containing multiple words, so it is necessary to model the interaction between words, which is not considered in word-level methods. Second, a phrase contains fewer words with less information compared to sentences, which prevents sentence encoders from taking the advantage of the ability of understanding full-length sentences. Thus, in this paper, we propose a novel cross-lingual phrase retriever named as XPR . Unlike previous cross-lingual retrieval methods that directly encode the input text, XPR produces phrase representations using example sentences, which can be collected from unlabeled text corpora. Initialized with a pretrained cross-lingual language 4193 model, XPR can either directly serve as an unsupervised retriever, or be further trained to produce better-aligned phrase representations. Besides, we propose the cross-lingual phrase contrast (XPCO ) loss for training XPR , where the model is trained to distinguish bilingual phrase pairs from negative examples. Furthermore, we create a cross-lingual phrase retrieval dataset, namely WikiXPR. WikiXPR contains 65K bilingual phrase pairs of eight language pairs, and provides example sentences for each phrase. We conduct a comprehensive evaluation of XPR on WikiXPR under four evaluation settings, i.e., unsupervised, supervised, zero-shot transfer, and multilingual supervised. Our XPR model substantially outperforms the retrieval baselines based on cross-lingual word embeddings and cross-lingual sentence encoders. XPR also shows impressive zero-shot transferability that enables the model to be trained in a language pair and directly perform phrase retrieval for other language pairs. Moreover, we present an in-depth analysis on XPR , showing that using example sentences improves both the learned XPR model and the phrase representations. Our contributions are summarized as follows:  We propose XPR , a novel cross-lingual phrase retriever that utilizes example sentences to produce phrase representations. We propose the cross-lingual phrase contrast loss for training XPR . We demonstrate the effectiveness of XPR on eight language pairs under four evaluation settings. We create a cross-lingual phrase retrieval dataset, which provides 65K bilingual phrase pairs with 4.2M example sentences in 8 language pairs. Cross-Lingual Retrieval Current cross-lingual text retrieval methods focus on word-level and sentence-level scenarios. Word-level cross-lingual retrieval methods typically train word embeddings on each language separately, and then align the word embeddings between languages by learning a mapping function (Mikolov et al., 2013; Dinu et al., 2014; Artetxe et al., 2016, 2018; Lample et al., 2018; Doval et al., 2018; Joulin et al., 2018). Similarly, cross-lingual sentence retrieval can be achieved by aligning sentence representations across different languages. LASER (Artetxe and Schwenk, 2019) learns a multilingual auto-encoder on multilingual parallel corpora to produce language-agnostic sentence embeddings. Training on parallel corpora, cross-lingual sentence representations can also be learned with neural machine translation (Schwenk, 2018), contrastive learning (Chi-dambaram et al., 2019; Feng et al., 2020; Chi et al., 2021b), translation span corruption (Chi et al., 2021a), or knowledge distillation (Ham and Kim, 2021). Thanks to the recent language model pretraining technique (Devlin et al., 2019), sentence encoders can be learned on a multilingual unlabeled text corpus without using parallel corpora (Conneau and Lample, 2019; Conneau et al., 2020; Chi et al., 2021c; Goswami et al., 2021). Phrase Retrieval Recent research on phrase retrieval typically learns phrase representations. Seo et al. (2019) propose to treat phrases as the smallest retrieval unit for open-domain question answering, where the phrases are encoded as indexable query-agnostic representations. The retrieval methods can be further improved with self-supervised pretraining, leading to better performance on open-domain question answering (Lee et al., 2021a,b). Additionally, DEER (Gillick et al., 2019) formulates the entity linking task as an entity phrase retrieval problem. However, these works study phrase retrieval in a monolingual scenario while we focus on cross-lingual phrase retrieval. Contrastive Learning Contrastive learning learns representations by a contrastive loss that encourages the positive data pairs to be more similar than other data pairs. It has shown to be effective for learning representations of a wide range of modalities including visual representations (He et al., 2020; Chen et al., 2020a; Grill et al., 2020), sentence representations (Kong et al., 2020; Chi et al., 2021b; Gao et al., 2021), audio representations (Saeed et al., 2021), etc. Different from previous work that performs contrastive learning at sentence level, we introduce contrastive learning to learn phrase representations. Figure 1 shows the overview of XPR . In this section, we first introduce the model architecture of XPR , and then present the cross-lingual phrase contrast loss. Finally, we show the training procedure of XPR . The model architecture of XPR is a Transformer (Vaswani et al., 2017) encoder shared across different languages. XPR can be initialized with pretrained cross-lingual language models, which have shown to produce well-aligned sentence representations (Hu et al., 2020; Chi et al., 2021b). Given a phrase p and an example sentence x = w 1 , . . . , w n with n tokens that contain the phrase. We denote the start and end indices of p as s and e , i.e., p = w s , . . . , w e . XPR first encodes x into a sequence of contextualized token representations 1 h 1 , . . . , h n = Transformer ( w 1 , . . . , w n ) . In general, a phrase can have more than one example sentence. Considering m example sentences X = x 1 , . . . , x m for the phrase p , XPR encodes the sentences separately, and uses the average of the phrase representations as the final phrase representation, i.e., (cid:80) x X x /m . Notice that XPR does not introduce additional parameters beyond the original Transformer encoder. Thus, after the initialization with a pretrained cross-lingual language model, XPR can directly serve as an unsupervised cross-lingual phrase retriever. Recent work (Chen et al., 2020a; Kong et al., 2020) has demonstrated the effectiveness of contrastive learning framework for learning visual and text representations. To learn language-agnostic phrase representations, we propose the cross-lingual phrase contrast (XPCO ) loss, where the goal is to distinguish the bilingual phrase pairs from negative examples. Formally, consider a mini-batch B = {P , Q} of bilingual phrase pairs, where P = { p } N and Q = { q } N stand for N phrases in a language and their translations in another language, respectively. For each phrase p  P , we sample example sentences X for p , and compute the phrase representation u as described in Section 3.1. Following Chen et al. (2020a), we apply a projection head over u that consists of two linear lay-4195 Algorithm 1 Training procedure of XPR Input: Bilingual phrase pair corpus D , unlabeled text corpus U , learning rate  , momentum coefficient  Output: XPR parameters  1: Initialize  ,  m 2: while not converged do 3: ( P , Q )  D 4: for i = 1 , 2 , . . . , N do 5: X  U s.t. p i  x ,  x  X 6: Y  U s.t. q i  y ,  y  Y 7: p i = f ( p i , X ;  ) 8: p  i = f ( p i , X ;  m ) 9: q i = f ( q i , Y ;  ) 10: q  i = f ( q i , Y ;  m ) 11: end for 12: g    LXPCO 13:      g 14:  m    m + (1   )  15: end while ers with a ReLU in between and a l 2 normalization followed. For simplicity, we denote the above operation that converts an input phrase p to a normalized vector as p = f ( p, X ;  ) , where  stands for the parameters of the encoder and the projection head. For each phrase q  Q , we employ a momentum encoder (He et al., 2020) to encode q : q  = f ( q, Y ;  m ) , where Y represents the example sentences of q , and  m represents the parameters of the momentum encoder. For the i -th phrase p i  P , q i  Q is its corresponding positive example and the other N  1 phrases are treated as negative examples. The contrastive loss in the direction of P  Q is defined as L ( P  Q ) =  N (cid:88) i =1 log exp( p (cid:62) i q  i /T ) (cid:80) Nj =1 exp( p (cid:62) i q  j /T ) (3) Similarly, we employ an additional contrastive loss in the direction of Q  P . The XPCO loss combines both directions, which is defined as LXPCO = L ( P  Q ) + L ( Q  P ) (4) where T is the softmax temperature. momentum encoder  m with a pretrained cross-lingual language model. For each training step, we first sample a mini-batch of bilingual phrase pairs ( P , Q ) from the bilingual phrase pair corpus D , and then sample example sentences X and Y for P and Q , respectively. Each example sentence x  X should contain the phrase p i , which is denoted as p i  x . With the phrase representations produced by the two encoders, we compute the XPCO loss, and update  with gradient descent. Notice that we do not perform back-propagation in the momentum encoder, which is learned by a momentum update (He et al., 2020) with a momentum coefficient of  . Given a phrase set P = { p } N with N candidate phrases , the goal is to find p  P with the same meaning of a query phrase q . With the trained XPR encoder  , we first sample example sentences candidate phrases and then compute their representations { p } N with f (  ;  ) . Then, for a query phrase q , we can find the corresponding phrase by:  p = arg max p i { p (cid:62) i q } (5) In practice, the representations of candidate phrases can be pre-computed for reuse. Moreover, although the example sentence number is limited during training, we can use more example sentences to obtain better phrase representation for retrieval. To evaluate our model, we create WikiXPR, a cross-lingual phrase retrieval dataset extracted from Wikipedia. WikiXPR consists of bilingual phrase pairs in eight English-centric language pairs, and contains large-scale example sentences for the phrases, which enable models to leverage contextual information to better understand phrases. In what follows, we describe how we construct the WikiXPR dataset. Manually translating phrases is expensive when building a large-scale bilingual phrase pair corpus. Therefore, we leverage the link information within Wikipedia for mining bilingual phrase pairs. Specifically, we first extract inter-language 4196 ar-en de-en es-en fr-en ja-en ko-en ru-en zh-en Total Train 4222 1931 1333 1315 14745 2138 5229 8326 39239 Dev 1408 644 445 438 4915 713 1743 2775 13081 Test 1407 644 445 438 4915 713 1743 2775 13080 Table 1: The number of bilingual phrase pairs for each language pair in WikiXPR. linked wiki entries from dbpedia 2 . We treat English as the pivot language, and choose a range of diverse languages to build our datasets, so that the models can be evaluated with different language families and scripts. We filter out time expressions, and the phrase pairs with low edit distance using ROUGE-L (Lin, 2004) as the distance measure. The phrase pairs with bidirectional ROUGE-L values higher than 0 . 5 are removed. In addition to phrase pairs in diverse languages, XPR also provides example sentences for each phrase, which aims to facilitate the research on phrase representation learning with example sentences. For each phrase, we retrieve example sentences from an unlabeled text corpus. In specific, we first extract raw sentences from Wikipedia dumps as our unlabeled text corpus. Then, we build sentence indices with the Elasticsearch 3 searching engine. For each phrase, we retain the searched sentences with at least 10 more characters than the phrase as the results. Besides, we only retain 32 example sentences for each phrase to keep a reasonable size for the resulting example sentence corpus. As shown in Table 1, we present the number of bilingual phrase pairs for each language pair in WikiXPR. The resulting WikiXPR dataset consists of 65,400 phrase pairs in eight language pairs, and 4.2M example sentences in total. For each phrase, WikiXPR provides 32 example sentences extracted from Wikipedia text. WikiXPR is split into training, dev, and test sets by 3:1:1, so WikiXPR can be used for diverse evaluation settings including the supervised setting, cross-lingual zero-shot transfer, etc. See detailed statistics in Appendix A. 2 downloads.dbpedia.org/2014/en/ 3 www.elastic.co 5 Experiments In this section, we first present four evaluation settings for cross-lingual phrase retrieval, and describe the models to be compared. Then, we present the experimental results. We conduct experiments on the cross-lingual phrase retrieval task on our WikiXPR dataset. Detailed description of WikiXPR can be found in Section 4. Since collecting or annotating parallel sentences can be expensive especially for low-resource languages, we only consider unlabeled text and the bilingual pairs provided by WikiXPR in our experiments. According to the difference in the training resource, we present the following four evaluation settings. Unsupervised Under the unsupervised setting, the retrieval model should not use any bilingual phrase pairs or other cross-lingual supervision such as bilingual dictionaries and parallel corpus. The language representations are typically learned from unlabeled text corpora. Supervised In the supervised setting, the retrieval model is trained on and tested on bilingual phrase pairs for each language pair separately, e.g., training and testing with English-French phrase pairs. Zero-Shot Transfer Zero-shot transfer is a widely-used setting in cross-lingual understanding tasks Conneau and Lample (2019); Wu and Dredze (2019), where models are trained in a source language but evaluated on other languages. We introduce this setting to the cross-lingual phrase retrieval task, e.g., training a model with English-French phrase pairs but performing retrieval between English and Chinese phrases. Multilingual Supervised In this setting, the retrieval model is able to use training data in multiple languages, e.g., training a model using a combined training set over all languages in WikiXPR and testing it for each language. Considering the lack of methods for cross-lingual phrase retrieval, we develop the following two baselines in our experiments: CLWE Cross-lingual word embeddings (CLWE ) encode words from various languages into a shared embedding space. For each word in a phrase, we first represent it with the pretrained fastText multilingual word vectors (Grave et al., 2018), and then map it to a shared embedding space via the VECMAP 4 (Artetxe et al., 2018) tool. Notice that VECMAP can be applied to both unsupervised and supervised scenarios. Finally, the retrieval is achieved by the nearest search using an average word vector as the phrase representation. CLSE Cross-lingual sentence encoders (CLSE ) produce language-agnostic sentence representations for the input text sequence. We use XLM-R base (Conneau et al., 2020) as the sentence encoder, which is pretrained on a large-scale multilingual text corpus. For the unsupervised setting, we use the averaged hidden vector from a specific middle layer as the phrase representation. For the other settings, we follow Wang et al. (2019), which learns an orthogonal mapping between the feature spaces of the training phrase pairs. As LASER (Artetxe and Schwenk, 2019) and LaBSE (Feng et al., 2020) utilize parallel corpora, we do not use them in our experiments. As for our model XPR described in Section 3, we initialize XPR with XLM-r base (Conneau et al., 2020) for a fair comparison. For each step, we use a batch of 256 phrase pairs and 4 example sen-4 github.com/artetxem/vecmap tences for each phrase. The model is optimized with the Adam (Kingma and Ba, 2015) optimizer with a learning rate of 2  10  5 for 100 epochs. The learning rate is scheduled with 1% warm-up steps and a linear decay during training. Unsupervised Results As present in Table 2, XPR obtains the best performance over all languages without any cross-lingual supervision, achieving an average accuracy@1 of 22.92. On the contrary, CLWE and CLSE only obtain 0.83 and 15.12, respectively. It indicates that XPR successfully leverage example sentences to produce better phrase representations. Besides, the performance varies in different languages. We observe that the retrieval between English and European languages can be easier than other language pairs when using CLSE and XPR . It is worth mentioning that CLWE and CLSE are proven to be effective for bilingual lexicon induction and cross-lingual sentence retrieval, respectively (Lample et al., 2018; Hu et al., 2020). Nonetheless, they do not perform as well as on word or sentence level tasks, indicating that they are not directly applicable to cross-lingual phrase retrieval. Supervised Results Under the supervised setting, XPR achieves an average accuracy of 83.94, largely outperforming the other two models over all evaluation language pairs. Comparing the results between the unsupervised and the supervised settings, all the three models greatly benefit from the training data. In particular, XPR pushes the average result from 7.34 to 87.32 for the en-ja phrase retrieval. The results suggest that the bilingual phrase pairs can help to learn cross-lingual alignment for both word-level and sentence-level representations. We find that using training data brings more gains for CLWE than CLSE , showing that the contextualized phrase representations in CLSE can be harder to align. Zero-shot Transfer In zero-shot transfer, the models are trained using an en-xx dataset but evaluated on all language pairs. The table only presents the results of the model trained on en-zh data. Detailed results of other transfer directions can be found in Appendix B. Although the XPR model only learns on en-zh training data, it performs surprisingly well on other languages. On en-es and en-ko, XPR even produces comparable results to the results in the supervised setting. Comparing the results to the unsupervised setting, XPR pushes the average accuracy from 22.92 to 76.99. This demonstrates the strong cross-lingual transferability of XPR , which allows our model to be applied to low-resource languages without training data. On the contrary, CLSE fails to leverage the en-zh training data for the retrieval in other languages, resulting in a consistent performance drop. Multilingual Supervised In the multilingual supervised setting, XPR obtains the best results over all models and settings, achieving an average accuracy of 88.57. Compared to the supervised setting, using the combined training data leads to consistent improvement over all languages, which demonstrates that XPR successfully leverage the supervision signals from both the same and different languages. We conduct ablation studies by removing main components from XPR . In specific, we compare three variants of XPR that are trained without example sentences, momentum update, or projection head, respectively. The evaluation results are shown in Table 3. Example Sentence We first investigate whether using example sentences helps cross-lingual phrase retrieval. During training, we remove the example sentences from XPR , i.e., the model extracts the phrase representation only from the input phrase itself. As shown in Table 3, removing example sentences substantially harms the performance of XPR for both the supervised and zero-shot transfer settings. Notice that example sentences are not parallel across languages, but they still make the resulting phrase representations from different languages better aligned. Besides, compared to the supervised setting, the gains are even larger for zero-shot transfer, improving the average accuracy from 60.48 to 78.33. The above results demonstrate that using example sentences not only learns better phrase representations, but also encourages cross-lingual alignment. age of the hidden vectors as the phrase representation. As shown in Table 3, the projection head provides consistent gains on the three language pairs, showing the effectiveness of the projection head in contrastive learning. The results also agree with the finding in visual representation learning (Chen et al., 2020a,b). Momentum Update We study the effects of momentum update used in XPR . It shows that the momentum update strategy slightly improves the results on all of the three evaluation language pairs, providing 0.84 accuracy improvement. We study the effects of the example sentence number used in XPR . We conduct an evaluation on the en-fr set of WikiXPR, under two settings where the example sentence number varies during training or inference: 1) Training and inference with various numbers of example sentences for each phrase, 2) Training with 32 example sentences for each phrase but inference with various numbers of example sentences. Figure 2 illustrates the evaluation results. It shows a trend that using more example sentences during inference notably improves the performance in both settings. The gain is larger when using fewer example sentences, demonstrating the effectiveness of using multiple example sentences for producing phrase representations. Comparing the results between the two settings, we find that the model moderately benefits from a large number of example sentences if we use a lower sentence number for inference. Although using more example sentences during training provides gains, the heavier computation load should be token into consideration. Recent work (Chi et al., 2021b,c) has shown that a middle layer can produce better-aligned sentence representations than the last layer, resulting in higher cross-lingual sentence retrieval performance. We investigate which hidden layer of XPR produces phrase representations that achieve higher retrieval accuracy. To this end, we evaluate XPR using representations from various hidden layers on the en-fr set of WikiXPR. As shown in Table 4, we present the evaluation results of XPR under both the unsupervised and the supervised settings. For the unsupervised XPR , we observe that Layer-11 produces the best results while the last layer even performs worse than the first layer. Differently, the supervised XPR obtains the best results on Layer-12, indicating that our XPCO loss encourages the model to fully utilize the last few layers. Moreover, it shows that using representations from higher layers of the supervised XPR leads to consistent improvement. We explore whether using momentum contrast (MOCO ; He et al. 2020) trains our XPR model better, which is proven to be effective for cross-lingual language model pretraining (Chi et al., 2021b). In specific, we train a variant of XPR with MOCO , which maintains more negative examples encoded by the momentum encoder with a queue with a length of 1024. The evaluation results are presented in Table 5. XPCO consistently outperforms MOCO on the three language pairs, suggesting that the negative examples stored in the queue can be out-of-date for contrastive learning. In this work, we propose a cross-lingual phrase retriever XPR , which outperforms the baseline retrievers on a range of diverse languages. Moreover, we create a cross-lingual phrase retrieval 4200 dataset that contains diverse languages with large-scale example sentences. For future work, we would like to improve XPR by: 1) extending XPR to asymmetric retrieval scenarios such as open-domain question answering, 2) exploring how to utilize parallel corpora for training XPR . XPR is designed as a cross-lingual phrase retriever that retrieve relevant phrases across different languages. We believe XPR would help the communication between the people who speak different languages. Besides, our work can facilitate the research on multilingual natural language processing (NLP), which helps to build NLP applications for low-resource languages. In addition, we construct the WikiXPR dataset using open-source data from Wikipedia and dbpedia. The work is supported by National Key R&D Plan (No. 2018YFB1005100), National Natural Science Foundation of China (No. U19B2020, 62172039, 61732005, 61602197 and L1924068), the funds of Beijing Advanced Innovation Center for Language Resources (No. TYZ19005), and in part by CCF-AFSG Research Fund under Grant No.RF20210005, and in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL). We would like to acknowledge Qian Liu for the helpful discussions.
abstain abstain abstain objective method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective method abstain method result abstain result objective objective objective method other other other other other other other other other other method other other method abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method method method method abstain method abstain abstain method abstain method method abstain method method method method abstain method abstain abstain other abstain method method abstain abstain abstain method abstain method abstain abstain result abstain abstain abstain abstain result method abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method method method method method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain objective method objective abstain method method method other other