It is well-known that distributional semantic approaches have difficulty in distinguishing between synonyms and antonyms (Grefen-stette, 1992; Pad and Lapata, 2003). Recent work has shown that supervision available in English for this task (e.g., lexical resources) can be transferred to other languages via crosslingual word embeddings. However, this kind of transfer misses monolingual distributional information available in a target language, such as contrast relations that are indicative of antonymy (e.g., hotwhilecold ). In this work, we improve the transfer by exploiting monolingual information, expressed in the form of co-occurrences with discourse markers that convey contrast. Our approach makes use of less than a dozen markers, which can easily be obtained for many languages. Compared to a baseline using only cross-lingual embeddings, we show absolute improvements of 4 10% F 1 -score in Vietnamese and Hindi. Recent work has shown that monolingual word embeddings in different languages can be aligned in an unsupervised manner (Artetxe et al., 2018; Kementchedjhieva et al., 2018). The resulting cross-lingual embeddings can be used to share supervision for lexical classification tasks across languages, when annotated data is not available in one language. For instance, a model for distinguishing lexical relations such as hypernymy and meronymy can be transferred to other languages (Glava and Vuli, 2018). However, this kind of transfer, using only cross-lingual embeddings, misses useful monolingual information available in the target language. In this paper, we consider one lexical classification task, namely the distinction between synonyms and antonyms, which is important English German hot cold new old ruhig kalt khl hitzig scharf hei kalt WOBEI hei (cold WHILE hot') co-occurrences with discourse markers Figure 1: Supervision for distinguishing antonyms from synonyms can be derived using discourse markers. for downstream applications such as contradiction detection (Harabagiu et al., 2006; Marneffe et al., 2008; Voorhees, 2008) and machine translation (Marton et al., 2011). To facilitate better transfer, we propose to use monolingual information in the form of word co-occurrences in contrast relations, in addition to cross-lingual embeddings (see Figure 1). In particular, we utilize the fact that discourse markers conveying contrast are more likely to be surrounded by antonyms than synonyms (e.g., hotwhilecold ), as shown by Roth and Schulte im Walde (2014). Our analysis reveals that (1) fine-grained semantic information that is required to distinguish synonyms from antonyms is insufficiently preserved cross-lingually in word embeddings, but (2) such information can be recovered (at least partially) by relying on linguistic intuitions about contrast relations in discourse. The identification of paradigmatic relations such as synonymy and antonymy has been a task of interest for more than a decade. Early work focused on the identification of instances of a single relation: for example, Charles and Miller (1989) investigated co-occurrences as an indicator for antonymy, Hearst (1992) introduced a pattern-based approach to identify hypernymy, and Edmonds and Hirst (2002) applied distributional statistics to identify synonymy. Beyond the identification of instances of a particular relation, more recent approaches attempt to distinguish relations such as synonymy and antonymy, based on lexico-syntactic patterns, distributional information, or combinations of both (Lin et al., 2003; Shwartz and Dagan, 2016). As supervision for weighting different features, most recent work makes use of lexical resources and/or lists of affix patterns that indicate contrast morphologically (Yih et al., 2012; Mohammad et al., 2013; Ono et al., 2015). Supervision for lexical classification tasks is not available in all languages. To overcome this difficulty, some approaches (Mrki et al., 2017; Glava and Vuli, 2018) combine resources for English and cross-lingual word embeddings to distinguish lexical relations in other languages. That is, they train and test one model across different languages. In contrast, we propose to first use unsupervised translation techniques to transfer supervision into a target language, given resources available in English. More specifically, our approach transfers supervision using a combination of unsupervised cross-lingual embeddings (Artetxe et al., 2018) and word co-occurrences with discourse markers that indicate contrast. In doing so, our approach generalizes the discourse-marker based model for relation classification by Roth and Schulte im Walde (2014) to a cross-lingual setting. In this work, we address the task of distinguishing synonyms and antonyms cross-lingually: given a word pair ( a; b ) in a target language, a model trained in a different source language determines whether this constitutes a synonym or an antonym pair. Our approach consists of two main ingredients: a translation module that creates training data in a target language (see Figure 2), and a classification module that uses this data in a supervised English corpus synonyms & antonyms  more,less  German corpus vecmap more less mehrkaum weniger cross-lingual distance(see3.1) n-best translations  mehr,mehr   kaum,weniger  re-ranking discoursecues (see 3.2) German train data  mehr,weniger  Figure 2: In our approach, we create training data in multiple languages by translating English data via cross-lingual word embeddings and discourse cues. fashion. Our focus is on the first part, which we describe in more detail below. For classification, we re-use the LexNet model by Shwartz and Dagan (2016), which is a feed-forward network that uses distributional word representations as input. Our transfer approach addresses a general setting that only assumes availability of monolingual data. Accordingly, we induce a translation dictionary of synonyms and antonyms in an unsupervised way, instead of relying on manually crafted dictionaries. We achieve this by finding n-best translations of English words in a cross-lingual word embedding space. We create this space using vecmap , a method that aligns and maps embedding spaces in two languages (Artetxe et al., 2018). Formally, we define the n th best translation w ( n ) t of a source word w s to be the n th nearest neighbor in the joint semantic space (based on cosine similarity). While 1 st best translations are often accurate, erroneous translations occur for example when two related words in a source language are close to the same words in the target language (e.g., less and more share the same nearest neighbor in Figure 2: mehr 'more'). In such cases, we aim to use monolingual information about lexical relations in the form of discourse cues. Specifically, given two antonymous words a s and b s (henceforth a s ? b s ) Language Corpus size Data size Ratio (in words) (in pairs) (#syn:#ant) English (train) 2.1 B 10 932 50:50 German (dev) 0.7 B 811 52:48 Hindi 0.5 B 1 000 50:50 Vietnamese 0.2 B 353 45:55 Table 1: Data statistics of the classification datasets as well as the corpora used to create word representations. in the source language, we want to choose translations a ( i ) t and b ( j ) t that are most likely antonymous as well. We achieve this by re-ranking translations a ( i ) t and b ( j ) t such that the probability Pr ( a ( i ) t ? b ( j ) t ) of them appearing in contrast relations is maximized. We only assume a set of discourse markers M that indicate contrast as knowledge about the target language. We approximate Pr ( a ( i ) t ? b ( j ) t ) by computing the pointwise mutual information (PMI) between two terms, conditioned on co-occurrence around a discourse marker m 2 M : Pr ( a ( i ) t ? b ( j ) t ) (cid:24) min m 2 M log N ( m ) n ( a ( i ) t ; b ( j ) t j m ) n ( a ( i ) t j m ) n ( b ( j ) t j m ) where N ( m ) is the frequency of discourse marker m in the corpus, n ( w j m ) is the number of times term w occurs with m , and n ( w 1 ; w 2 j m ) is the number of co-occurrences w 1  m  w 2 . For simplicity, we count all joint occurrences within a sentence, regardless of sentence length and distance between words. In practice, the set M is constructed by manually translating eight discourse markers that frequently indicate contrast relations according to the Penn Discourse Treebank 2.0 (Prasad et al., 2007): although , by comparison , by contrast , however , nevertheless , nonetheless , though , and thus . Our experiments test the performance of an off-the-shelf system for lexical relation classification under different cross-lingual settings. In particular, we evaluate the performance of unsupervised cross-lingual word embeddings and assess the benefits of translation and re-ranking, taking into account word co-occurrences in contrast relations. Vietnamese. The antonymy and synonymy dataset by Nguyen et al. (2017) is used for training and estimating an upper bound in English. For crosslingual development and hyper-parameter selection, we use the German dataset by Glava and Vuli (2018). Specifically, we select the number of hidden layers {0, 1}, learning rate {0.001,,0.1}, dropout rate {0.0, 0.5}, and whether to include information on paths between words {yes, no}. 1 As examples of under-resourced languages, we evaluate on the Vietnamese dataset ViCon by Nguyen et al. (2018) and on a new Hindi dataset, which we crawled from hindi2dictionary.com . 2 Note that annotated data in these languages is held out exclusively for testing. The task addressed here is to distinguish synonymous from antonymous words. Accordingly, we consider only word pairs for training and testing that are marked as antonyms or synonyms, even if the original dataset also contains unrelated words or other relations. Statistics of all considered word pairs, the ratio between synonyms and antonyms, as well as sizes of the text corpora used in our experiments are given in Table 1. Models. We test our proposed approach against two baselines: NoTrans and BestTrans . NoTrans simply uses the training data in English and allows us to test in how far semantic properties related to the distinction between synonymy and antonymy are preserved cross-lingually. BestTrans uses 1 st best translations of the English training data in a target language, as described in Section 3.1. Our own approach, henceforth called LingTrans , is based on n-best translations and exploits word co-occurrences in contrast relations for reranking, as described in Section 3.2. All three models take cross-lingual word embeddings as input. We created these as follows. First, the unsupervised morphological analyzer of Xu et al. (2018) is used to lemmatize the monolingual corpora for the respective languagesthe German Wikipedia, the Vietnamese portion of the LORELEI pack (Strassel and Tracey, 2016), and the HindEnCorp corpus (Bojar et al., 2014). This step ensures that we can compare the (stems of) 1 Since we do not assume access to syntactic paths in the target languages, we only experimented with lexical surface paths and found them to consistently degrade performance. 2 To verify the correctness of the crawled data, we asked a native speaker to validate all instances. word tokens in text to those of the word forms that actually appear in the synonymy/antonymy datasets. Note that, while we use an unsupervised morphological analyzer, a stemmer can also be used, if available for that language. Next, we created monolingual embeddings for each language using fastText (Bojanowski et al., 2017). Finally, we applied the unsupervised variant of vecmap (Artetxe et al., 2018) to compute alignments and cross-lingual mappings. The word and discourse marker co-occurrence counts required for our approach are computed on the same monolingual corpora used for training monolingual embeddings. Results. Table 2 shows macro-averaged F 1 scores for crosslingual synonymy/antonymy distinction (top part) as well as monolingual results in English for comparison (bottom part). The monolingual results show that word embeddings provide appropriate information for classification in most instances, achieving an F 1 -score of 70.1. The comparatively low cross-lingual results by NoTrans indicate that aligning and mapping embedding spaces does not preserve all semantic properties relevant to the distinction between synonymy and antonymy. The use of first-best translations in BestTrans alleviates this issue partially and improves F 1 by up to 4.6 points. Our intuition regarding discourse cues in LingTrans leads to further improvements of up to 9.9 additional points in F 1 and considerably closes the gap between monolingual and cross-lingual results (56.8 (cid:0) 64.2 vs. 70.1 F 1 ). Based on an approximate randomization test over the respective test items (Yeh, 2000), we find the improvements of our proposed approach LingTrans over NoTrans to be significant in Vietnamese (p < 0.01), German and Hindi (p < 0.1). In contrast, there is no significant difference in performance between BestTrans and NoTrans , confirming that both translation and reranking are required to achieve consistent gains. We examine how far our proposed approach affects performance on the task of synonymyantonymy distinction and discuss remaining shortcomings. A first observation concerns the general variance of results across languages. In addition to differences in terms of available corpus sizes, we observe different challenges in each language and dataset. Notably, each dataset is created with its own linguistically motivated definitions of antonymy and synonymy, some of which are more relaxed than others. For example, Nguyen et al. (2018) consider words which are strongly associated but highly dissimilar to each other as antonyms and words that are highly similar in meaning as synonyms. The performance of our baseline NoTrans suggests that the distinction between synonyms and antonyms in the crosslingual space is harder for Vietnamese than for the other languages. At least partially, this is due to errors and noise from preprocessing (i.e., lemmatization). Also, Vietnamese belongs to a different language family than English. Consequently, information relevant to the synonymyantonymy distinction may be distributed differently in the target language space than in the source language. In Hindi, some errors specific to the data involve synonyms that denote the same mythological deity (e.g.,  / Hanuman/Bajrangbali') and thus require factual knowledge for classification. Even for our development language German, results suggest that there is room for improvement. We examine the improvements achieved and remaining errors in German in more detail below. Classification improvements. The main improvement of our LingTrans approach over BestTrans is reflected in an increased number of correctly classified antonyms, mostly including antonymous adjectives (e.g., stndig steadily' vs. sporadisch sporadically') but also mutually exclusive nouns (e.g., Privatgelnde private grounds' vs. ffentlichkeit public'). Compared to the NoTrans baseline, we find BestTrans to substantially increase the number of correctly classified synonyms (by almost 20%). In particular, this affects words derived from the same stem ( unzhlig/zahllos countless'), synonymous verbs ( pfuschen/schummeln cheat') as well as (near-)synonymous adjectives ( verfgbar/verwendbar available/usable'). Translation improvements. The observed quantitative improvements are in line with qualitative improvements that we see in the automatically generated training data. For example, more and less have the same 1 st -best translation in German according to the cross-lingual space ( mehr more'); as illustrated in Figure 2, reranking is required to find the correct translation for the latter word (i.e., weniger less'). To quantify translation improvements of our proposed reranking method, we count the number of words overlapping between the automatically translated data and data available in the target languages. Compared to the BestTrans baseline, we find our LingTrans approach to improve overlap for German, Hindi, and Vietnamese by 2%, 5% and 5%, respectively. Despite the increase in word overlap, it is worth noting that the overlap between training and testing in terms of actual data instances (i.e., pairs of words) remains constant between 0% and 1%. A partial explanation for the improved results could thus be that our reranking approach lets us find translations that are generally more likely to have an antonym (or synonym)regardless of which other word they are presented with. A similar observation was made by Levy et al. (2015) regarding hypernymy classification, in which they found some words to be generally more likely to be category words. Whether similar biases exist in synonymyantonymy classification will be subject of future work. Remaining errors. We categorized a randomly selected subset of 30 mis-classifications. We found 30% of the errors to be related to sparsity. In these cases, at least one word was infrequent or the word pair never co-occurred in the corpus (e.g., unvershnbar/unvereinbar un-forgiving/irreconcilable'). 27% of the errors can be attributed to problems with the test data because it contains synonyms that are only indirectly related (e.g., parieren/ausweichen parry/dodge'). morphologically marked antonyms (e.g., Richtigkeit correctness' vs. Unrichtigkeit incorrectness') and instances of synonyms that involve two identical words (e.g., Verwandtschaft/Verwandtschaft kinship'), which presumably is an artifact of the automatic translation step used in creating the German dataset (see Glava and Vuli (2018)). Both types of errors could be identified by simple heuristics but are incorrectly classified by the model. On the other hand, hard cases involve words of different linguistic register, which are expected to be distributed differently and therefore hard to capture by distributional methods (e.g., mittellos/verarmt penniless/impoverished'). In summary, we find substantial improvement through our transfer method, compared to the baselines. However, further improvements could be achieved, for example, taking into account regis-ter (e.g., formal vs. colloquial) and morphological marking (e.g., negation affixes). We proposed to combine unsupervised crosslingual embeddings and discourse cues to generate supervision for distinguishing synonyms and antonyms in under-resourced languages. Compared to a baseline that uses only cross-lingual embeddings, we showed that the use of a small set of discourse markers indicating contrast can yield absolute improvements of up to 10% F 1 -score. The simplicity of our approach allows to easily incorporate other features (e.g., morphological marking) and it can be extended to further languages or lexical relations. For example, discourse markers that indicate specification and instantiation relations (e.g. specifically , for instance ) could be used to detect hypernymy (cf. Roth and Schulte im Walde (2014)). Beyond classification, another direction for future work is to extend our approach to distinguish synonyms and antonyms from unrelated word pairs. An interesting direction to pursue would be to use multiple related languages, to aid lexical relation classification in an under-resourced language, instead of transferring supervision from a single language (English). This research was supported by the German Research Foundation (DFG EXC 285), and by Contract HR0011-15-2-0025 and Agreement HR0011-15-2-0023 with the US DARPA.
abstain abstain abstain objective result result abstain abstain abstain abstain objective abstain objective abstain objective other other other other other other other objective abstain abstain method method abstain method method method method method method method abstain method other method method abstain method method abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective abstain abstain objective abstain other