Word translation or bilingual lexicon induction (BLI) is a key cross-lingual task, aiming to bridge the lexical gap between different languages. In this work, we propose a robust and effective two-stage contrastive learning framework for the BLI task. At Stage C1, we propose to refine standard cross-lingual linear maps between static word embeddings (WEs) via a contrastive learning objective; we also show how to integrate it into the self-learning procedure for even more refined cross-lingual maps. In Stage C2, we conduct BLI-oriented contrastive fine-tuning of mBERT, unlocking its word translation capability. We also show that static WEs induced from the C2-tuned' mBERT complement static WEs from Stage C1. Comprehensive experiments on standard BLI datasets for diverse languages and different experimental setups demonstrate substantial gains achieved by our framework. While the BLI method from Stage C1 already yields substantial gains over all state-of-the-art BLI methods in our comparison, even stronger improvements are met with the full two-stage framework: e.g., we report gains for 112/112 BLI setups, spanning 28 language pairs. Bilingual lexicon induction (BLI) or word translation is one of the seminal and long-standing tasks in multilingual NLP (Rapp, 1995; Gaussier et al., 2004; Heyman et al., 2017; Shi et al., 2021, inter alia ). Its main goal is learning translation correspondences across languages, with applications of BLI ranging from language learning and acquisition (Yuan et al., 2020; Akyurek and Andreas, 2021) to machine translation (Qi et al., 2018; Duan et al., 2020; Chronopoulou et al., 2021) and the development of language technology in low-resource languages and domains (Irvine and Callison-Burch, 2017; Heyman et al., 2018). A large body of recent BLI work has focused on the so-called mapping-based methods (Mikolov et al., 2013; Artetxe et al., mBERT/mT5 Static Word Embeddings f 1 ( y n i ) f 1 ( x m i ) f 1 ( x j ) f 2 ( y n i ) f 2 ( x j ) f 2 ( x m i ) Seed Dictionary C1 Alignment C2 Alignment  (1  ) Procrustes Attract RepelNegative Sampling Figure 1: An illustration of the proposed two-stage BLI approach (see 2). 2018; Ruder et al., 2019). 1 Such methods are particularly suitable for low-resource languages and weakly supervised learning setups: they support BLI with only as much as few thousand word translation pairs (e.g., 1k or at most 5k) as the only bilingual supervision (Ruder et al., 2019). 2 Unlike for many other tasks in multilingual NLP (Doddapaneni et al., 2021; Chau and Smith, 2021; Ansell et al., 2021), state-of-the-art (SotA) BLI results are still achieved via static word embeddings (WEs) (Vulic et al., 2020b; Liu et al., 2021b). A typical modus operandi of mapping-based approaches is to first train monolingual WEs independently on monolingual corpora and then map them to a shared cross-lingual space via linear (Mikolov et al., 2013; 1 They are also referred to as projection-based or alignment-based methods (Glava et al., 2019; Ruder et al., 2019). 2 In the extreme, fully unsupervised mapping-based BLI methods can leverage monolingual data only without any bilingual supervision (Lample et al., 2018; Artetxe et al., 2018; Hoshen and Wolf, 2018; Mohiuddin and Joty, 2019; Ren et al., 2020, inter alia ). However, comparative empirical analyses (Vulic et al., 2019) show that, with all other components equal, using seed sets of only 500-1,000 translation pairs, always outperforms fully unsupervised BLI methods. Therefore, in this work we focus on this more pragmatic (weakly) supervised BLI setup (Artetxe et al., 2020); we assume the existence of at least 1,000 seed translations per each language pair. Glava et al., 2019) or non-linear mapping functions (Mohiuddin et al., 2020). In order to achieve even better results, many BLI methods also apply a self-learning loop where training dictionaries are iteratively (and gradually) refined, and improved mappings are then learned in each iteration (Artetxe et al., 2018; Karan et al., 2020). However, there is still ample room for improvement, especially for lower-resource languages and dissimilar language pairs (Vulic et al., 2019; Nasution et al., 2021). On the other hand, another line of recent research has demonstrated that a wealth of lexical semantic information is encoded in large multilingual pretrained language models (LMs) such as mBERT (Devlin et al., 2019), but 1) it is not straightforward to transform the LMs into multilingual lexical encoders (Liu et al., 2021b), 2) extract word-level information from them (Vulic et al., 2020b, 2021), and 3) word representations extracted from these LMs still cannot surpass static WEs in the BLI task (Vulic et al., 2020b; Zhang et al., 2021). Motivated by these insights, in this work we investigate the following research questions: (RQ1) Can we further improve (weakly supervised) mapping-based BLI methods based on static WEs? (RQ2) How can we extract more useful crosslingual word representations from pretrained multilingual LMs such as mBERT or mT5? Inspired by the wide success of contrastive learning techniques in sentence-level representation learning (Reimers and Gurevych, 2019; Carlsson et al., 2021; Gao et al., 2021), we propose a two-stage contrastive learning framework for effective word translation in (weakly) supervised setups; it leverages and combines multilingual knowledge from static WEs and pretrained multilingual LMs. Stage C1 operates solely on static WEs: in short, it is a mapping-based approach with self-learning, where in each step we additionally fine-tune linear maps with contrastive learning that operates on gradually refined positive examples (i.e., true translation pairs), and hard negative samples. Stage C2 fine-tunes a pretrained multilingual LM (e.g., mBERT), again with a contrastive learning objective, using positive examples as well as negative examples extracted from the output of C1. Finally, we extract word representations from the multilingual LM fine-tuned in Stage C2, and combine them with static cross-lingual WEs from Stage C1; the combined representations are then used for BLI. We run a comprehensive set of BLI experiments on the standard BLI benchmark (Glava et al., 2019), comprising 8 diverse languages, in several setups. Our results indicate large gains over state-of-the-art BLI models: e.g.,  +8 Precision@1 points on average, +10 points for many language pairs, gains for 107/112 BLI setups already after Stage C1 (cf., RQ1), and for all 112/112 BLI setups after Stage C2 (cf., RQ2 and RQ3). Moreover, our findings also extend to BLI for lower-resource languages from another BLI benchmark (Vulic et al., 2019). Finally, as hinted in recent work (Zhang et al., 2021), our findings validate that multilingual lexical knowledge in LMs, when exposed and extracted as in our contrastive learning framework, can complement the knowledge in static cross-lingual WEs (RQ3), and benefit BLI. We release the code and share the data at: https: //github.com/cambridgeltl/ContrastiveBLI . Preliminaries and Task Formulation. In BLI, we assume two vocabularies X = { w x 1 , . . . , w x |X| } and Y = { w y 1 , . . . , w y |Y| } associated with two respective languages L x and L y . We also assume that each vocabulary word is assigned its (static) type-level word embedding (WE); that is, the respective WE matrices for each vocabulary are X  R |X| d , Y  R |Y| d . Each WE is a d -dim row vector, with typical values d =300 for static WEs (e.g., fastText) (Bojanowski et al., 2017), and d =768 for mBERT. 3 We also assume a set of seed translation pairs D 0 = { ( w xm 1 , w yn 1 ) , ..., ( w xm | D 0 | , w yn | D 0 | ) } for training (Mikolov et al., 2013; Glava et al., 2019), where 1  m i  |X | , 1  n i  |Y| . Typical values for the seed dictionary size |D 0 | are 5 k pairs and 1 k pairs (Vulic et al., 2019), often referred to as supervised (5k) and semi-supervised or weakly supervised settings (1k) (Artetxe et al., 2018). Given another test lexicon DT = { ( w xt 1 , w yg 1 ) , ..., ( w xt |D T | , w yg |D T | ) } , where D 0  DT =  , for each L x test word w xt i in DT the goal is to retrieve its correct translation from L y 's vocabulary Y , and evaluate it against the gold L y translation w yg i from the pair. Method in a Nutshell. We propose a novel 3 We also tried XLM ( d =1 , 280 ) and mT5 small ( d =512 ); mBERT is the best-performing pretrained LM in our preliminary investigation. two-stage contrastive learning (CL) method, with both stages C1 and C2 realised via contrastive learning objectives (see Figure 1). Stage C1 (2.1) operates solely on static WEs, and can be seen as a contrastive extension of mapping-based BLI approaches with static WEs. In practice, we blend contrastive learning with the standard SotA mapping-based framework with self-learning: VecMap (Artetxe et al., 2018), with some modifi-cations. Stage C1 operates solely on static WEs in exactly the same BLI setup as prior work, and thus it can be evaluated independently. In Stage C2 (2.2), we propose to leverage pretrained multilingual LMs for BLI: we contrastively fine-tune them for BLI and extract static decontextualised' WEs from the tuned LMs. These LM-based WEs can be combined with WEs obtained in Stage C1 (2.3). Stage C1 is based on the VecMap framework (Artetxe et al., 2018) which features 1) dual linear mapping , where two separate linear transformation matrices map respective source and target WEs to a shared cross-lingual space; and 2) a self-learning procedure that, in each iteration i refines the training dictionary and iteratively improves the mapping. We extend and refine VecMap's self-learning for supervised and semi-supervised settings via CL. Initial Advanced Mapping. After (cid:96) 2 -normalising word embeddings, 4 the two mapping matrices, denoted as W x for the source language L x and W y for L y , are computed via the Advanced Mapping (AM) procedure based on the training dictionary, as fully described in Appendix A.1; while VecMap leverages whitening, orthogonal mapping, re-weighting and de-whitening operations to derive mapped WEs, we compute W x and W y such that a one-off matrix multiplication produces the same result (see Appendix A.1 for the details). Contrastive Fine-Tuning. At each iteration i , after the initial AM step, the two mapping matrices W x and W y are then further contrastively fine-tuned via the InfoNCE loss (Oord et al., 2018), a standard and robust choice of a loss function in CL research (Musgrave et al., 2020; Liu et al., 2021c,b). The core idea is to attract' aligned WEs of positive examples (i.e., true translation pairs) coming from the dictionary D i  1 , and repel' hard negative samples , that is, words which are semantically similar 4 Unlike VecMap, we do not mean-center WEs as this yielded slightly better results in our preliminary experiments. These hard negative samples are extracted as follows. Let us suppose that ( w xm i , w yn i ) is a translation pair in the current dictionary D i  1 , with its constituent words associated with static WEs x m i , y n i  R 1  d . We then retrieve the nearest neighbours of y n i W y from XW x and derive  w xm i  X ( w xm i excluded) , a set of hard negative samples of size N neg . In a similar (symmetric) manner, we also derive the set of negatives  w yn i  Y ( w yn i ex-cluded). We use  D to denote a collection of all hard negative set pairs over all training pairs in the current iteration i . We then fine-tune W x and W y by optimising the following contrastive objective: s i,j = exp(cos( x i W x , y j W y ) / ) , (1) p i = s m i ,n i (cid:80) w yj { w yni } (cid:83)  w yni s m i ,j + (cid:80) w xj   w xmi s j,n i , (2) min W x , W y  E ( w xmi ,w yni ) D CL log( p i ) . denotes a standard temperature parameter. The objective, formulated here for a single positive example, spans all positive examples from the current dictionary, along with the respective sets of negative examples computed as described above. Self-Learning. The application of (a) initial mapping via AM and (b) contrastive fine-tuning can be repeated iteratively. Such self-learning loops typically yield more robust and better-performing BLI methods (Artetxe et al., 2018; Vulic et al., 2019). At each iteration i , a set of automatically extracted high-confidence translation pairs D add are added to the seed dictionary D 0 , and this dictionary D i = D 0  D add is then used in the next iteration i + 1 . Our dictionary augmentation method slightly deviates from the one used by VecMap. We leverage the most frequent N freq source and target vocabulary words, and conduct forward and backward dictionary induction (Artetxe et al., 2018). Unlike VecMap, we do not add stochasticity to the process, and simply select the top N aug high-confidence 4355 word pairs from forward (i.e., source-to-target) induction and another N aug pairs from the backward induction. In practice, we retrieve the 2  N aug pairs with the highest Cross-domain Similarity Local Scaling (CSLS) scores (Lample et al., 2018), 5 remove duplicate pairs and those that contradict with ground truth in D 0 , and then add the rest into D add . For the initial AM step, we always use the augmented dictionary D 0  D add ; the same augmented dictionary is used for contrastive fine-tuning in weakly supervised setups. 6 We repeat the self-learning loop for N iter times: in each iteration, we optimise the contrastive loss NCL times; that is, we go NCL times over all the positive pairs from the training dictionary (at this iteration). N iter and NCL are tunable hyper-parameters. Self-learning in Stage C1 is summarised in Algorithm 1. 2.2 Stage C2 Previous work tried to prompt off-the-shelf multilingual LMs for word translation knowledge via masked natural language templates (Gonen et al., 2020), averaging over their contextual encodings in a large corpus (Vulic et al., 2020b; Zhang et al., 2021), or extracting type-level WEs from the LMs directly without context (Vulic et al., 2020a, 2021). However, even sophisticated templates and WE extraction strategies still typically result in BLI performance inferior to fastText (Vulic et al., 2021). (BLI-Oriented) Contrastive Fine-Tuning. Here, we propose to fine-tune off-the-shelf multilingual LMs relying on the supervised BLI signal: the aim is to expose type-level word translation knowledge directly from the LM, without any external corpora. In practice, we first prepare a dictionary of positive examples for contrastive fine-tuning: (a) DCL = D 0 when |D 0 | spans 5 k pairs, or (b) when |D 0 | =1 k , we add the N aug =4 k automatically extracted highest-confidence pairs from Stage C1 (based on their CSLS scores, not present in D 0 ) to D 0 (i.e., DCL spans 1 k + 4 k word pairs). We then extract N neg hard negatives in the same way as in 2.1, relying on the shared cross-lingual space derived as the output of Stage C1. Our hypothesis is that a difficult task of discerning between true translation pairs and highly similar non-translations as hard negatives, formulated within a contrastive 5 Further details on the CSLS similarity and its relationship to cosine similarity are available in Appendix A.2. 6 When starting with 5k pairs, we leverage only D 0 for contrastive fine-tuning, as D add might deteriorate the quality of the 5k-pairs seed dictionary due to potentially noisy input. learning objective, will enable mBERT to expose its word translation knowledge, and complement the knowledge already available after Stage C1. Throughout this work, we assume the use of pretrained mBERT base model with 12 Transformer layers and 768 -dim embeddings. Each raw word input w is tokenised, via mBERT's dedicated tokeniser, into the following sequence: [ CLS ][ sw 1 ] . . . [ sw M ][ SEP ] , M  1 , where [ sw 1 ] . . . [ sw M ] refers to the sequence of M constituent subwords/WordPieces of w , and [ CLS ] and [ SEP ] are special tokens (Vulic et al., 2020b). The sequence is then passed through mBERT as the encoder, its encoding function denoted as f  (  ) : it extracts the representation of the [CLS] token in the last Transformer layer as the representation of the input word w . The full set of mBERT's parameters  then gets contrastively fine-tuned in Stage C2, again relying on the InfoNCE CL loss: s (cid:48) i,j = exp(cos( f  ( w xi ) , f  ( w yj )) / ) , (4) p (cid:48) i = s (cid:48) m i ,n i (cid:80) w yj { w yni } (cid:83)  w yni s (cid:48) m i ,j + (cid:80) w xj   w xmi s (cid:48) j,n i , (5) min   E ( w xmi ,w yni ) D CL log( p (cid:48) i ) . Type-level WE for each input word w is then obtained simply as f  (cid:48) ( w ) , where  (cid:48) refers to the parameters of the BLI-tuned' mBERT model. In order to combine the output WEs from Stage C1 and the mBERT-based WEs from Stage C2, we also need to map them into a shared' space: in other words, for each word w , its C1 WE and its C2 WE can be seen as two different views of the same data point. We thus learn an additional linear orthogonal mapping from the C1-induced cross-lingual WE space into the C2-induced cross-lingual WE space. It transforms (cid:96) 2 normed 300-dim C1-induced cross-lingual WEs into 768 -dim cross-lingual WEs. Learning of the linear map W  R d 1  d 2 , where in our case d 1 =300 and d 2 =768 , is formulated as a Generalised Procrustes problem (Schnemann, 1966; Viklands, 2006) operating on all (i.e., both L x and L y ) words from the seed translation dictionary D 0 . 7 7 Technical details of the learning procedure are described in Appendix A.3. It is important to note that in this case we do not use word translation pairs ( w xm i , w yn i ) directly to learn the mapping, but rather each word w xm i and w yn i is duplicated to create training pairs ( w xm i , w xm i ) and ( w yn i , w yn i ) , where the left word/item in each pair is assigned its WE from C1, and the right word/item is assigned its WE after C2. Unless noted otherwise, a final representation of an input word w is then a linear combination of (a) its C1-based vector v w mapped to a 768 dim representation via W , and (b) its 768 -dim encoding f  (cid:48) ( w ) from BLI-tuned mBERT: (1   ) v w W (cid:107) v w W (cid:107) 2 +  f  (cid:48) ( w ) (cid:107) f  (cid:48) ( w ) (cid:107) 2 , (7) where  is a tunable interpolation hyper-parameter. Monolingual WEs and BLI Setup. We largely follow the standard BLI setup from prior work (Artetxe et al., 2018; Joulin et al., 2018; Glava et al., 2019; Karan et al., 2020, inter alia ). The main evaluation is based on the standard BLI dataset from Glava et al. (2019): it comprises 28 language pairs with a good balance of typologically similar and distant languages (Croatian: HR , English: EN , Finnish: FI , French: FR , German: DE , Italian: IT , Russian: RU , Turkish: TR ). Again following prior work, we rely on monolingual fastText vectors trained on full Wikipedias for each language (Bojanowski et al., 2017), where vocabularies in each language are trimmed to the 200K most frequent words (i.e., |X | =200 k and |Y| =200 k ). The same fastText WEs are used for our Stage C1 and in all baseline BLI models. mBERT in Stage C2 operates over the same vocabularies spanning 200 k word types in each language. We use 1k translation pairs (semi-supervised BLI mode) or 5k pairs (supervised) as seed dictionary D 0 ; test sets span 2 k pairs (Glava et al., 2019). With 56 BLI directions in total, 8 this yields a total of 112 BLI setups for each model in our comparison. The standard Precision@1 (P@1) BLI measure is reported, and we rely on CSLS ( k =10 ) to score word similarity (Lample et al., 2018). 9 Training Setup and Hyperparameters. Since standard BLI datasets typically lack a validation set (Ruder et al., 2019), following prior work (Glava et al., 2019; Karan et al., 2020) we conduct hyper-parameter tuning on a single, randomly selected language pair EN  TR, and apply those hyperpa-rameter values in all other BLI runs. both for L i  L j and L j  L i directions. 9 The same trends in results are observed with Mean Reciprocal Rank (MRR) as another BLI evaluation measure (Glava et al., 2019); we omit MRR scores for clarity. Moreover, similar relative trends, but with slightly lower absolute BLI scores, are observed when replacing CSLS with the simpler cosine similarity measure: the results are available in the Appendix. In Stage C1, when |D 0 | =5 k , the hyperparam-eter values are N iter =2 , NCL =200 , N neg =150 , N freq =60 k , N aug =10 k . SGD optimiser is used, with a learning rate of 1 . 5 and  =0 . 99 . When |D 0 | =1 k , the values are N iter =3 , NCL =50 , N neg =60 , N freq =20 k , and N aug =6 k ; SGD with a learning rate of 2 . 0 ,  =1 . 0 . =1 . 0 and dropout is 0 in both cases, and the batch size for contrastive learning is always equal to the size of the current dictionary |D CL | (i.e., |D 0 | (5k case), or |D 0 D add | which varies over iterations (1k case); see 2.1). In Stage C2, N neg =28 and the maximum sequence length is 6 . We use AdamW (Loshchilov and Hut-ter, 2019) with learning rate of 2 e  5 and weight decay of 0 . 01 . We fine-tune mBERT for 5 epochs, with a batch size of 100 ; dropout rate is 0 . 1 and  =0 . 1 . Unless noted otherwise,  is fixed to 0 . 2 . Baseline Models. Our BLI method is evaluated against four strong SotA BLI models from recent literature, all of them with publicly available implementations. Here, we provide brief summaries: 10 RCSLS (Joulin et al., 2018) optimises a relaxed CSLS loss, learns a non-orthogonal mapping, and has been established as a strong BLI model in empirical comparative analyses as its objective function is directly BLI-oriented' (Glava et al., 2019). VecMap 's core components (Artetxe et al., 2018) have been outlined in 2.1. LNMap (Mohiuddin et al., 2020) non-linearly maps the original static WEs into two latent semantic spaces learned via non-linear autoencoders, 11 and then learns another non-linear mapping between the latent autoencoder-based spaces. FIPP (Sachidananda et al., 2021), in brief, first finds common (i.e., isomorphic) geometric structures in monolingual WE spaces of both languages, and then aligns the Gram matrices of the WEs found in those common structures. For all baselines, we have verified that the hy-perparameter values suggested in their respective repositories yield (near-)optimal BLI performance. Unless noted otherwise, we run VecMap, LNMap, and FIPP with their own self-learning procedures. 12 10 For further technical details and descriptions of each BLI model, we refer to their respective publications. We used publicly available implementations of all the baseline models. 11 This step is directed towards mitigating anisomorphism (Sgaard et al., 2018; Dubossarsky et al., 2020) between the original WE spaces, which should facilitate their alignment. 12 RCSLS is packaged without self-learning; extending it to support self-learning is non-trivial and goes beyond the scope of this work. Model Variants. We denote the full two-stage BLI model as C2 (Mod) , where Mod refers to the actual model/method used to derive the shared crosslingual space used by Stage C2. For instance, C2 (C1) refers to the model variant which relies on our Stage C1, while C2 (RCSLS) relies on RCSLS as the base method. We also evaluate BLI performance of our Stage C1 BLI method alone. Multilingual LMs. We adopt mBERT as the default pretrained multilingual LM in Stage C2. Our supplementary experiments also cover the 1280 dim XLM model 13 (Lample and Conneau, 2019) and 512 -dim mT5 small (Xue et al., 2021). 14 For clarity, we use C2 [LM] to denote C2 (C1) obtained from different LMs; when [LM] is not spec-ified, mBERT is used. We adopt a smaller batch size of 50 for C2 [XLM] considering the limit of GPU memory, and train C2 [mT5] with a larger learning rate of 6 e  4 for 6 epochs, since we found it much harder to train than C2 [mBERT] . The main results are provided in Table 1, while the full results per each individual language pair, and also with cosine similarity as the word retrieval function, are provided in Appendix E. The main findings are discussed in what follows. Stage C1 versus Baselines. First, we note that there is not a single strongest baseline among the four SotA BLI methods. For instance, RCSLS and VecMap are slightly better than LNMap and FIPP with 5k supervision pairs, while FIPP and VecMap come forth as the stronger baselines with 1k supervision. There are some score fluctuations over individual language pairs, but the average performance of all baseline models is within a relatively narrow interval: the average performance of all four baselines is within 3 P@1 points with 5k pairs (i.e., ranging from 38.22 to 41.22), and VecMap, FIPP, and LNMap are within 2 points with 1k pairs. Strikingly, contrastive learning in Stage C1 already yields substantial gains over all four SotA BLI models, which is typically much higher than the detected variations between the baselines. We mark that C1 improves over all baselines in 51/56 BLI setups (in the 5k case), and in all 56/56 BLI setups when D 0 spans 1k pairs. The average gains 13 We pick the XLM large model pretrained on 100 languages with masked language modeling (MLM) objective. with the C1 variant are  5 P@1 points over the SotA baselines with 5k pairs, and  6 P@1 points with 1k pairs (ignoring RCSLS in the 1k scenario). Note that all the models in comparison, each currently considered SotA in the BLI task, use exactly the same monolingual WEs and leverage exactly the same amount of bilingual supervision. The gains achieved with our Stage C1 thus strongly indicate the potential and usefulness of word-level contrastive fine-tuning when learning linear crosslingual maps with static WEs (see RQ1 from 1). Stage C1 + Stage C2. The scores improve further with the full two-stage procedure. The C2 (C1) BLI variant increases average P@1 by another 3.3 4358 [1k] Pairs BG  CA CA  HE HE  BG VecMap 39.43 24.64 31.55 FIPP 34.29 20.63 26.38 C1 41.88 30.56 33.49 mBERT 1.64 1.28 0.88 mBERT (tuned) 13.90 3.43 4.76 C2 (C1) 44.28 33.99 37.78 [1k] Pairs ET  HU HU  EU EU  ET VecMap 35.55 20.03 9.83 FIPP 30.30 11.58 8.22 C1 40.35 20.09 13.00 mBERT 15.40 16.97 23.70 mBERT (tuned) 20.59 22.30 28.62 C2 (C1) 44.64 28.26 21.35 C2 (C1,  = 0 . 4 ) -34.62 36.70 Table 2: BLI scores on the Panlex-BLI sets. (5k) and 3 P@1 points (1k), and we observe gains for all language pairs in both translation directions, rendering Stage C2 universally useful. These gains indicate that mBERT does contain word translation knowledge in its parameters. However, the model must be fine-tuned (i.e., transformed) to unlock' the knowledge from its parameters: this is done through a BLI-guided contrastive fine-tuning procedure (see 2.2). Our findings thus further confirm the rewiring hypothesis' from prior work (Vulic et al., 2021; Liu et al., 2021b; Gao et al., 2021), here validated for the BLI task (see RQ2 from 1), which states that task-relevant knowledge at sentenceand word-level can be rewired'/exposed from the off-the-shelf LMs, even when leveraging very limited task supervision, e.g., with only 1k or 5k word translation pairs as in our experiments. Performance over Languages. The absolute BLI scores naturally depend on the actual source and target languages: e.g., the lowest absolute performance is observed for morphologically rich ( HR , RU , FI , TR ) and non-Indo-European languages ( FI , TR ). However, both C1 and C2 (C1) mode variants offer wide and substantial gains in performance for all language pairs, irrespective of the starting absolute score. This result further suggests wide applicability and robustness of our BLI method. Evaluation on Lower-Resource Languages. The robustness of our BLI method is further tested on another BLI evaluation set: PanLex-BLI (Vulic et al., 2019), which focuses on BLI evaluation for lower-resource language; 1k training pairs and 2k test pairs are derived from PanLex (Kamholz et al., 2014). The results for a subset of six languages (Basque: EU , Bulgarian: BG , Catalan: CA , Estonian: ET , Hebrew: HE , Hungarian: HU ) are [5k] Pairs DE  TR TR  HR HR  RURCSLS 30.99 24.60 37.19 C2 (RCSLS) 36.52 33.17 44.77 VecMap 27.18 25.99 37.98 C2 (VecMap) 34.95 34.29 44.98 C1 34.69 32.37 41.66 C2 (C1) 38.86 36.32 46.40 [1k] Pairs DE  TR TR  HR HR  RURCSLS 18.21 13.84 24.72 C2 (RCSLS) 25.40 22.52 33.88 VecMap 23.37 20.50 36.09 C2 (VecMap) 27.91 26.84 40.45 C1 32.03 27.00 39.40 C2 (C1) 34.85 32.16 42.14 Table 3: Stage C2 with different support' methods: RCSLS, VecMap, and C1. presented in Table 2. Overall, the results further confirm the efficacy of the C2 (C1) , with gains observed even with typologically distant language pairs (e.g., HE  BG and EU  ET ). Usefulness of Stage C2? The results in Table 1 have confirmed the effectiveness of our two-stage C2 (C1) BLI method (see RQ3 in 1). However, Stage C2 is in fact independent of our Stage C1, and thus can also be combined with other standard BLI methods. Therefore, we seek to validate whether combining exposed mBERT-based translation knowledge can also aid other BLI methods. In other words, instead of drawing positive and negative samples from Stage C1 (2.2) and combining C2 WEs with WEs from C1 (2.3), we replace C1 with our baseline models. The results of these C2 (RCSLS) and C2 (VecMap) BLI variants for a selection of language pairs are provided in Table 3. The gains achieved with all C2 (  ) variants clearly indicate that Stage C2 produces WEs which aid all BLI methods. In fact, combining it with RC-4359 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 10152025303540455055606570 A cc u r ac y ( P @ 1 ) DE TR TR HR HR RU RU IT IT FR 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 10152025303540455055606570 A cc u r ac y ( P @ 1 ) DE TR TR HR HR RU RU IT IT FR 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0510152025303540455055606570 A cc u r ac y ( P @ 1 ) BG CA CA HE HE BG ET HU HU EU EU ET Figure 2: BLI scores with different  values: (left) |D 0 | = 5 k ; (middle) |D 0 | = 1 k ; (right) PanLex-BLI, |D 0 | = 1 k . SLS and VecMap yields even larger relative gains over the base models than combining it with our Stage C1. However, since Stage C1 (as the base model) performs better than RCSLS and VecMap, the final absolute scores with C2 (C1) still outperform C2 (RCSLS) and C2 (VecMap) . Different Multilingual LMs? Results on eight language pairs, shown in Table 4, indicate that C2 (C1) is also compatible with different LMs. The overall trend is that all three C2 [LM] variants derive some gains when compared to C1. C2 [mBERT] is the best-performing model and derives gains in all 112 / 112 BLI setups (also see Appendix E); C2 [mT5] outperforms C1 in all 16 / 16 cases, and the gains are observed for 14 / 16 cases with C2 [XLM] . It is also worth noticing that C2 [XLM] can surpass C2 [mBERT] on several pairs. Combining C1 and C2? The usefulness of combining the representations from two stages is measured through varying the value of  for several BLI setups. The plots are shown in Figure 2, and indicate that Stage C1 is more beneficial to the performance, with slight gains achieved when allowing the influx' of mBERT knowledge (e.g.,  in the [0 . 0  0 . 3] interval). While mBERT-based WEs are not sufficient as standalone representations for BLI, they seem to be even more useful in the combined model for lower-resource languages on PanLex-BLI, with steeper increase in performance, and peak scores achieved with larger  -s. Ablation Study , with results summarised in Table 5, displays several interesting trends. First, both CL and self-learning are key components in the 1k-setups: removing any of them yields substantial drops. In 5k-setups, self-learning becomes less important, and removing it yields only negligible drops, while CL remains a crucial component (see also Appendix F). Further, Table 5 complements the results from Figure 2 and again indicates that, while Stage C2 indeed boosts word translation capacity of mBERT, using mBERT features alone is still not sufficient to achieve competitive [5k] Pairs EN  DE  IT  C1 w/o CL 41.58 39.30 42.67 C1 w/o SL 50.99 45.07 48.39 C1 51.31 46.14 48.92 mBERT 9.55 9.39 8.13 mBERT (tuned) 15.87 18.66 20.18 C1 + mBERT 51.55 46.25 48.91 C2 (C1) 54.31 48.86 51.91 [1k] Pairs EN  DE  IT  C1 w/o CL 39.46 37.54 40.37 C1 w/o SL 39.31 32.59 36.45 C1 47.16 43.94 46.55 mBERT 9.55 9.39 8.13 mBERT (tuned) 17.29 20.92 23.29 C1 + mBERT 47.56 44.08 46.74 C2 (C1) 49.84 46.61 49.22 Table 5: Ablation study. After all, pretrained LMs are contex-tualised encoders designed for (long) sequences rather than individual words or tokens. Finally, Table 5 shows the importance of fine-tuning mBERT before combining it with C1-based WEs (2.3): directly adding WEs extracted from the off-the-shelf mBERT does not yield any benefits (see the scores for the C1+mBERT variant, where  is also 0 . 2 ). On the other hand, contrastive fine-tuning reshapes the subspaces towards a shared (cross-lingual) space, the effects of which are then also reflected in mBERT's improved BLI capability (see Table 5 again). To understand the role of CL in Stage C1, we visualise static WEs mapped by C1 without CL (i.e., AM+SL, see 2.1) and also from the complete Stage C1, respectively. Figure 4 shows that C1 without CL already learns a sensible cross-lingual space. However, we note that advanced mapping (AM) in C1 without CL learns a (near-)orthogonal map, which might result in mismatches, especially with dissimilar language pairs. With TR-HR, the plot reveals that there exists a gap between C1-aligned WE spaces although the final BLI performance still gets improved: this might be due to repelling' negatives from each other during CL. This work is related to three topics, each with a large body of work; we can thus provide only a condensed summary of the most relevant research. Mapping-Based BLI. These BLI methods are highly popular due to reduced bilingual supervision requirements; consequently, they are applicable to low-resource languages and domains, learning linear (Lample et al., 2018; Artetxe et al., 2018; Joulin et al., 2018; Patra et al., 2019; Jawanpuria et al., 2019; Sachidananda et al., 2021) and non-linear maps (Mohiuddin et al., 2020; Glava and Vulic, 2020; Ganesan et al., 2021), typically using self-learning in weakly supervised setups. Contrastive Learning in NLP aims to learn a semantic space such that embeddings of similar text inputs are close to each other, while repelling' dissimilar ones. It has shown promising performance on training generic sentence encoders (Giorgi et al., 2021; Carlsson et al., 2021; Liu et al., 2021a; Gao et al., 2021) and downstream tasks like summarisation (Liu and Liu, 2021) or NER (Das et al., 2021). Exposing Lexical Knowledge from Pretrained LMs. Extracting lexical features from off-the-shelf multilingual LMs typically yields subpar performance in lexical tasks (Vulic et al., 2020b). To unlock the lexical knowledge encoded in PLMs, Liu et al. (2021a) and Vulic et al. (2021) fine-tune LMs via contrastive learning with manually cu-rated or automatically extracted phrase/word pairs to transform it into effective text encoders. Wang et al. (2021) and Liu et al. (2021c) apply similar techniques for phrase and word-in-context representation learning, respectively. The success of these methods suggests that LMs store a wealth of lexical knowledge: yet, as we confirm here for BLI, fine-tuning is typically needed to expose it. We have proposed a simple yet extremely effective and robust two-stage contrastive learning framework for improving bilingual lexicon induction (BLI). In Stage C1, we tune cross-lingual linear mappings between static word embeddings with a contrastive objective and achieve substantial gains in 107 out of 112 BLI setups on the standard BLI benchmark. In Stage C2, we further propose a contrastive fine-tuning procedure to harvest crosslingual lexical knowledge from multilingual pretrained language models. The representations from this process, when combined with Stage C1 embeddings, have resulted in further boosts in BLI performance, with large gains in all 112 setups. We have also conducted a series of finer-grained evaluations, analyses and ablation studies. (cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70)(cid:70) We thank the anonymous reviewers for their valuable feedback. This work is supported by the ERC PoC Grant MultiConvAI (no. 957356) and a research donation from Huawei. YL and FL are supported by Grace & Thomas C. H. Chan Cambridge International Scholarship. Our research aims to benefit the efforts in delivering truly multilingual language technology also to under-resourced languages and cultures via bridging the lexical gap between languages, groups and cultures. As a key task in cross-lingual NLP, bilingual lexicon induction or word translation has broad applications in, e.g., machine translation, language acquisition and potentially protecting endangered languages. Furthermore, compared with many previous studies, we stress the importance of diversity in the sense that our experiments cover various language families and include six lower-resource languages from the PanLex-BLI dataset. Hoping that our work can contribute to extending modern NLP techniques to lower-resource and under-represented languages, we focus on semi-supervised settings and achieve significant improvements with self-learning techniques. The two BLI datasets we use are both publicly available. To our best knowledge, the data (i.e., word translation pairs) do not contain any sensitive information and have no foreseeable risk.
abstain objective objective method result objective abstain abstain other other abstain abstain abstain abstain abstain abstain method abstain abstain abstain other objective abstain method abstain abstain abstain method abstain result abstain method other abstain method method abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain method method method method method abstain abstain other abstain abstain abstain abstain abstain abstain method method method method abstain method other abstain other abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain method method abstain method method method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method method abstain abstain method method method method method method method abstain method method method abstain method method method method abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain other result abstain abstain result abstain abstain abstain result abstain abstain other abstain abstain result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain result result method other other other other other other other other method objective objective objective abstain method other other other abstain abstain method abstain method abstain