English verbs have multiple forms. For instance, talk may also appear as talks , talked or talking , depending on the context. The NLP task of lemmatization seeks to map these diverse forms back to a canonical one, known as the lemma. We present a simple joint neural model for lemmatization and morphological tagging that achieves state-of-the-art results on 20 languages from the Universal Dependencies corpora. Our paper describes the model in addition to training and decoding procedures. Error analysis indicates that joint morphological tagging and lemmatization is especially helpful in low-resource lemmatization and languages that display a larger degree of morphological complexity. Code and pre-trained models are available at https://sigmorphon.github.io/ sharedtasks/2019/task2/ . Lemmatization is a core NLP task that involves a string-to-string transduction from an inflected word form to its citation form, known as the lemma. More concretely, consider the English sentence: The bulls are running in Pamplona . A lemmatizer will seek to map each word to a form you may find in a dictionaryfor instance, mapping running to run . This linguistic normalization is important in several downstream NLP applications, especially for highly inflected languages. Lemmatization has previously been shown to improve recall for information retrieval (Kanis and Skorkovsk, 2010; Monz and De Rijke, 2001), to aid machine translation (Fraser et al., 2012; Chahuneau et al., 2013) and is a core part of modern parsing systems (Bjrkelund et al., 2010; Zeman et al., 2018). instance, in the sentence A running of the bulls took place in Pamplona , the word running is its own lemma, since, here, running is a noun rather than an inflected verb. Several counter-examples exist to this trend, as discussed in depth in Haspelmath and Sims (2013). Thus, a good lemmatizer must make use of some representation of each word's sentential context. The research question in this work is, then, how do we design a lemmatization model that best extracts the morpho-syntax from the sentential context? Recent work (Bergmanis and Goldwater, 2018) has presented a system that directly summarizes the sentential context using a recurrent neural network to decide how to lemmatize. As Bergmanis and Goldwater (2018)'s system currently achieves state-of-the-art results, it must implicitly learn a contextual representation that encodes the necessary morpho-syntax, as such knowledge is requisite for the task. We contend, however, that rather than expecting the network to implicitly learn some no       families happy All are similar to each other POS = (cid:9) CASE = (cid:9)(cid:10)(cid:8) NUM = (cid:11)(cid:7) GEN = (cid:5)(cid:4)(cid:8) POS = (cid:1) CASE = (cid:9)(cid:10)(cid:8) NUM = (cid:11)(cid:7) POS = (cid:9) CASE = (cid:9)(cid:10)(cid:8) NUM = (cid:11)(cid:7) POS = (cid:3) CASE = (cid:9)(cid:10)(cid:8) NUM = (cid:11)(cid:7) POS = (cid:9) CASE = (cid:9)(cid:10)(cid:8) NUM = (cid:12)(cid:6) POS = (cid:11) POS = (cid:9) CASE = (cid:1)(cid:2)(cid:2) NUM = (cid:12)(cid:6) Morph. tion of morpho-syntax, it is better to explicitly train a joint model to morphologically disambiguate and lemmatize. Indeed, to this end, we introduce a joint model for the introduction of morphology into a neural lemmatizer. A key feature of our model is its simplicity : Our contribution is to show how to stitch existing models together into a joint model, explaining how to train and decode the model. However, despite the model's simplicity, it still achieves a significant improvement over the state of the art on our target task: lemmatization. Experimentally, our contributions are threefold. First, we show that our joint model achieves state-of-the-art results, outperforming (on average) all competing approaches on a 20-language subset of the Universal Dependencies (UD) corpora (Nivre et al., 2017). Second, by providing the joint model with gold morphological tags, we demonstrate that we are far from achieving the upper bound on performanceimprovements on morphological tagging could lead to substantially better lemmatization. Finally, we provide a detailed error analysis indicating when and why morphological analysis helps lemmatization. We offer two tangible recommendations: one is better off using a joint model (i) for languages with fewer training data available and (ii) languages that have richer morphology. Our system and pre-trained models on all languages in the latest version of the UD corpora 12 are released at https://sigmorphon.github. io/sharedtasks/2019/task2/ . 1 We compare to previously published numbers on nonrecent versions of UD, but the models we release are trained on the current version (2.3). Most languages (Dryer and Haspelmath, 2013) in the world exhibit a linguistic phenomenon known as inflectional morphology , which causes word forms to mutate according to the syntactic category of the word. The syntactic context in which the word form occurs determines which form is properly used. One privileged form in the set of in-flections is called the lemma . We regard the lemma as a lexicographic convention, often used to better organize dictionaries. Thus, the choice of which inflected form is the lemma is motivated by tradition and convenience, e.g., the lemma is the infinitive for verbs in some Indo-European languages, rather than by linguistic or cognitive concerns. Note that the stem differs from the lemma in that the stem may not be an actual inflection. 3 In the NLP literature, the syntactic category that each inflected form encodes is called the morphological tag . The morphological tag generalizes traditional part-of-speech tags, enriching them with further linguistic knowledge such as tense, mood, and grammatical case. We call the individual keyattribute pairs morphological attributes . An example of a sentence annotated with morphological tags and lemmata in context is given in Figure 2. The task of mapping a sentence to a sequence of morphological tags is known as morphological tagging . Notation. Let w = w 1 , . . . , w n be a sequence of n words. Each individual word is denoted as w i . Likewise, let m = m 1 , . . . , m n and (cid:96) = (cid:96) 1 , . . . , (cid:96) n be sequences of morphological tags and lemmata, respectively. We will denote the set of all tags seen in a treebank as Y . We remark that m i is w i 's morphological tag (e.g. [ POS = N , CASE = NOM , NUM = SG ] as a single label) and 3 The stem is also often ill-defined. (cid:96) i is w i 's lemma. We will denote a language's discrete alphabet of characters as  . Thus, we have w i , (cid:96) i    . Furthermore, we c = c 1 , . . . , c n be a vector of characters where c i   . The primary contribution of this paper is a joint model of morphological tagging and lemmatization. The intuition behind the joint model is simple: high-accuracy lemmatization requires a representation of the sentential context, in which the word occurs (this behind has been evinced in 1)a morphological tag provides the precise summary of the context required to choose the correct lemma. Armed with this, we define our joint model of lemmatization and morphological tagging as: p ( (cid:96) , m | w ) = p ( (cid:96) | m , w ) p ( m | w ) (1) =   n (cid:89) i =1 p ( (cid:96) i | m i , w i ) (cid:124) (cid:123)(cid:122) (cid:125) Neural Transducer   p ( m | w ) (cid:124) (cid:123)(cid:122) (cid:125) Neural Tagger Figure 1 illustrates the structure of our model in the form of a graphical model. We will discuss the lemmatization factor and the morphological tagging factor following two subsections, separately. We caution the reader that the discussion of these models will be brief: Neither of these particular components is novel with respect to the literature, so the formal details of the two models is best found in the original papers. The point of our paper is to describe a simple manner to combine these existing parts into a state-of-the-art lemmatizer. We employ a simple LSTM-based tagger to recover the morphology of a sentence (Heigold et al., 2017; Cotterell and Heigold, 2017). We also experimented with the neural conditional random field of Malaviya et al. (2018), but Heigold et al. (2017) gave slightly better tagging scores on average and is faster to train. Given a sequence of n words w = w 1 , . . . , w n , we would like to obtain the morphological tags m = m 1 , . . . , m n for each word, where m i  Y . The model first obtains a word representation for each token using a character-level biLSTM (Graves et al., 2013) embedder, which is then input to a word-level biLSTM tagger that predicts tags for each word. Given a function cLSTM that returns the last hidden state of a character-based LSTM, first we obtain a word representation u i for word w i as, u i = [ cLSTM ( c 1 . . . c n ); cLSTM ( c n . . . c 1 )] (2) where c 1 , . . . , c n is the character sequence of the word. This representation u i is then input to a word-level biLSTM tagger. The word-level biLSTM tagger predicts a tag from Y . A full description of the model is found in Heigold et al. (2017). We use standard cross-entropy loss for training this model and decode greedily while predicting the tags during test-time. Note that greedy decoding is optimal in this tagger as there is no interdependence between the tags m i . Neural sequence-to-sequence models (Sutskever et al., 2014; Bahdanau et al., 2015) have yielded state-of-the-art performance on the task of generating morphological variantsincluding the lemmaas evinced in several recent shared tasks on the subject (Cotterell et al., 2016, 2017, 2018). Our lemmatization factor in eq. (1) is based on such models. Specifically, we make use of a hard-attention mechanism (Xu et al., 2015; Rastogi et al., 2016), rather than the original soft-attention mechanism. Our choice of hard attention is motivated by the performance of Makarov and Clematide (2018)'s system at the CoNLL-SIGMORPHON task. We use a nearly identical model, but opt for an exact dynamic-programming-based inference scheme (Wu et al., 2018). 4 We briefly describe the model here. Given an inflected word w and a tag m , we would like to obtain the lemma (cid:96)    , dropping the subscript for simplicity. Moreover, for the remainder of this section the subscripts will index into the character string (cid:96) , that is (cid:96) = (cid:96) 1 , . . . , (cid:96) | (cid:96) | , where each (cid:96) i   . A character-level biLSTM encoder embeds w to h (enc) . The decoder LSTM produces h (dec) j , reading the concatenation of the embedding of the previous character (cid:96) j  1   and the tag embedding h (tag) , which is produced by an order-invariant linear function. In contrast to soft attention, hard attention models the alignment distribution explicitly. We denote A = { a 1 , . . . , a | w | } | (cid:96) | as the set of all monotonic alignments from w to (cid:96) where an alignment aligns each target character (cid:96) j to exactly one source character in w and for a  A , a j = i 4 Our formulation differs from the work of Wu et al. (2018) in that we enforce monotonic hard alignments, rather than allow for non-monotonic alignments. p ( (cid:96) | m, w ) = (cid:88) a  A p ( (cid:96), a | m, w ) (3) = (cid:88) a  A | (cid:96) | (cid:89) j =1 p ( (cid:96) j | a j , (cid:96) <j , m, w ) (4) p ( a j | a j  1 , (cid:96) <j , m, w ) = (cid:88) a  A | (cid:96) | (cid:89) j =1 p ( (cid:96) j | h (enc) a j , h (dec) j ) (5) p ( a j | a j  1 , h (enc) , h (dec) j ) The summation is computed with dynamic programmingspecifically, using the forward algorithm for hidden Markov models (Rabiner, 1989). p ( (cid:96) j | h (enc) a j , h (dec) j ) is a two-layer feed-forward network followed by a softmax. The transition p ( a j | a j  1 , h (enc) , h (dec) j ) is the multiplicative attention function with h (enc) and h (dec) j as input. To enforce monotonicity, p ( a j | a j  1 ) = 0 if a j < a j  1 . We consider two manners, by which we decode our model. The first is a greedy decoding scheme. The second is a crunching (May and Knight, 2006) scheme. We describe each in turn. Note that we slightly abuse notation since the argmax here is approximate : exact decoding of our neural lemmatizer is hard. This sort of scheme is also referred to as pipeline decoding. Crunching. In the crunching scheme, we first extract a k -best list of taggings from the morphological tagger. For an input sentence w , call the k -best tags for the i th word K ( w i ) . Crunching then says we should decode in the following manner (cid:96) (cid:63)i = argmax (cid:96) (8) log (cid:88) m i K ( w i ) p ( (cid:96) | m i , w i ) p ( m i | w ) Crunching is a tractable heuristic that approximates true joint decoding 5 and, as such, we expect it to outperform the more nave greedy approach. In our model, a simple application of maximum-likelihood estimation (MLE) is unlikely to work well. The reason is that our model is a discriminative directed graphical model (as seen in Figure 1) and, thus, suffers from exposure bias (Ranzato et al., 2015). The intuition behind the poor performance of MLE is simple: the output of the lemmatizer depends on the output of the morphological tagger; as the lemmatizer has only ever seen correct morphological tags, it has never learned to adjust for the errors that will be made at the time of decoding. To compensate for this, we employ jackknifing (Agic and Schluter, 2017), which is standard practice in many NLP pipelines, such as dependency parsing. Jackknifing for training NLP pipelines is quite similar to the oft-employed cross-validation. We divide our training data into  splits. Then, for each split i  { 1 , . . . ,  } , we train the morphological tagger on the i th split, and then decode it, using either greedy decoding or crunching, on the remaining (   1) splits. This technique helps avoid exposure bias and improves the lemmatization performance, which we will demonstrate empirically in 4. Indeed, the model is quite ineffective without this training regime. Note that we employ jackknifing for both the greedy decoding scheme and the crunching decoding scheme. To enable a fair comparison with Bergmanis and Goldwater (2018), we use the Universal Dependencies Treebanks (Nivre et al., 2017) for all our experiments. Following previous work, we use v2.0 of the treebanks for all languages, except Dutch, for which v2.1 was used due to inconsistencies in v2.0. The standard splits are used for all treebanks. 5 True joint decoding would sum over all possible morphological tags, rather than just the k -best. While this is tractable in our setting in the sense that there are, at, most, 1662 morphological tags (in the case of Basque), it is significantly slower than using a smaller k . Indeed, the probability distribution that morphological taggers learn tend to be peaked to the point that considering improbable tags is not necessary. For the morphological tagger, we use the baseline implementation from Malaviya et al. (2018). This implementation uses an input layer and linear layer dimension of 128 and a 2-layer LSTM with a hidden layer dimension of 256. The Adam (Kingma and Ba, 2014) optimizer is used for training and a dropout rate (Srivastava et al., 2014) of 0.3 is enforced during training. The tagger was trained for 10 epochs. For the lemmatizer, we use a 2-layer biLSTM encoder and a 1-layer LSTM decoder with 400 hidden units. The dimensions of character and tag embedding are 200 and 40, respectively. We enforce a dropout rate of 0.4 in the embedding and encoder LSTM layers. The lemmatizer is also trained with Adam and the learning rate is 0.001. We halve the learning rate whenever the development log-likelihood increases and we perform early-stopping when the learning rate reaches 1  10  5 . We apply gradient clipping with a maximum gradient norm of 5. Lematus. The current state of the art is held by Bergmanis and Goldwater (2018), who, as discussed in 1, provide a direct context-to-lemma approach, avoiding the use of morphological tags. We remark that Bergmanis and Goldwater (2018) assume a setting where lemmata are annotated at the token level, but morphological tags are not available; we contend, however, that such a setting is not entirely realistic as almost all corpora annotated with lemmata at the token level include morpho-syntactic annotation, including the vast majority of the UD corpora. Thus, we do not consider it a stretch to assume the annotation of morphological tags to train our joint model. 6 UDPipe. 6 After correspondence with Toms Bergmanis, we would like to clarify this point. While Bergmanis and Goldwater (2018) explores the model in a token-annotated setting, as do we, the authors argue that such a model is better for a very low-resource scenario where the entire sentence is not annotated for lemmata. We concede this pointour current model is not applicable in such a setting. However, we note that a semi-supervised morphological tagger could be trained in such a situation as well, which may benefit lemmatization. performs lemmatization using an averaged perceptron tagger that predicts a (lemma rule, UPOS) pair. Here, a lemma rule generates a lemma by removing parts of the word prefix/suffix and prepending and appending a new prefix/suffix. A guesser first produces correct lemma rules and the tagger is used to disambiguate from them. Lemming. The strongest non-neural baseline we consider is the system of Mller et al. (2015), who, like us, develop a joint model of morphological tagging lemmatization. In contrast to us, however, their model is globally normalized (Lafferty et al., 2001). Due to their global normalization, they directly estimate the parameters of their model with MLE without worrying about exposure bias. However, in order to efficiently normalize the model, they heuristically limit the set of possible lemmata through the use of edit trees (Chrupaa, 2008), which makes the computation of the partition function tractable. Morfette. Much like Mller et al. (2015), Morfette relies on the concept of edit trees. However, a simple perceptron is used for classification with hand-crafted features. A full description of the model is given in Chrupaa et al. (2008). Experimentally, we aim to show three points. i) Our joint model (eq. (1)) of morphological tagging and lemmatization achieves state-of-the-art accuracy; this builds on the findings of Bergmanis and Goldwater (2018), who show that context significantly helps neural lemmatization. Moreover, the upper bound for contextual lemmatizers that make use of morphological tags is much higher, indicating room for improved lemmatization with better morphological taggers. ii) We discuss a number of error patterns that the model seems to make on the languages, where absolute accuracy is lowest: Latvian, Estonian and Arabic. We suggest possible paths forward to improve performance. iii) We offer an explanation for when our joint model does better than the context-to-lemma baseline. We show through a correlational study that our joint approach with morphological tagging helps the most in two cases: low-resource languages and morphologically rich languages. The first experiment we run focuses on pure performance of the model. Our goal is to determine whether joint morphological tagging and lemmatization improves average performance in a state-of-the-art neural model. Evaluation Metrics. For measuring lemmatization performance, we measure the accuracy of guessing the lemmata correctly over an entire corpus. To demonstrate the effectiveness of our model in utilizing context and generalizing to unseen word forms, we follow Bergmanis and Goldwater (2018) and also report accuracies on tokens that are i) ambiguous , i.e., more than one lemmata exist for the same inflected form, ii) unseen , i.e., where the inflected form has not been seen in the training set, and iii) seen unambiguous , i.e., where the inflected form has only one lemma and is seen in the training set. Results. The results showing comparisons with all other methods are summarized in Figure 3. Each bar represents the average accuracy across 20 languages. Our method achieves an average accuracy of 95 . 42 and the strongest baseline, Bergmanis and Goldwater (2018), achieves an average accuracy of 95 . 05 . The difference in performance ( 0 . 37 ) is statistically significant with p < 0 . 01 under a paired permutation test. We outperform the strongest baseline in 11 out of 20 languages and underperform in only 3 languages with p < 0 . 05 . The difference between our method and all other baselines is statistical significant with p < 0 . 001 in all cases. We highlight two additional features of the data. First, decoding using gold morphological tags gives an accuracy of 98 . 04 for a difference in performance of +2 . 62 . We take the large difference between the upper bound and the current performance of our model to indicate that improved morphological tagging is likely to significantly help lemmatization. Second, it is noteworthy that training with gold tags, but decoding with predicted tags, yields performance that is significantly worse than every baseline except for UDPipe. This speaks for the importance of jackknifing in the training of joint morphological tagger-lemmatizers that are directed and, therefore, suffer from exposure bias. In Figure 4, we observed crunching further im-A r a b i c B a s qu e C r oa t i a n D u t c h E s t o n i a n F i nn i s h G er m a n G ree k H i nd i H un ga r i a n I t a li a n L a t v i a n P o li s h P o r t u g u e s e R o m a n i a n R u ss i a n S l ova k S l ov e n i a n T u r k i s h U r du -0.05% 0.00% 0.05% 0.10% 0.15% 0.20% k = 40 k = 20 k = 10 k = 5 Figure 4: Relative improvement on validation set with crunching over greedy decoding for different values of k . proves performance of the greedy decoding scheme. In 8 out of 20 languages, the improvement is statistical significant with p < 0 . 05 . We select the best k for each language based on the development set. In Figure 5, we provide a language-wise breakdown of the performance of our model and the model of Bergmanis and Goldwater (2018). Our strongest improvements are seen in Latvian, Greek and Hungarian. When measuring performance solely over unseen inflected forms, we achieve even stronger gains over the baseline method in most languages. This demonstrates the generalization power of our model beyond word forms seen in the training set. In addition, our accuracies on ambiguous tokens are also seen to be higher than the baseline on average, with strong improvements on highly inflected languages such as Latvian and Russian. Finally, on seen unambiguous tokens, we note improvements that are similar across all languages. We attempt to identify systematic error patterns of our model in an effort to motivate future work. For this analysis, we compare predictions of our model and the gold lemmata on three languages with the weakest absolute performance: Estonian, Latvian and Arabic. First, we note the differences in the average lengths of gold lemmata in the tokens we guess incorrectly and all the tokens in the corpus. The lemmata we guess incorrectly are on average 1.04 characters longer than the average length of all the lemmata in the corpus. We found that the length of the incorrect lemmata does not correlate strongly with their frequency. Next, we identify the most common set of edit operations in each language that would transform the incorrect hypothesis to the gold lemma. This set of edit operations was lang # tokens # tags ours Lematus  arabic 202000 349 93.1 93.55 -0.48 basque 59700 884 96.74 96.55 0.2 croatian 146000 1105 96.16 95.7 0.48 dutch 163000 62 97.26 97.65 -0.4 estonian 17000 482 85.83 84.99 0.99 finnish 135000 1669 94.79 94.31 0.51 german 227000 683 97.46 97.72 -0.26 greek 36500 346 95.29 94.22 1.13 hindi 261000 939 98.88 98.92 -0.05 hungarian 16700 580 96.13 94.99 1.2 italian 236000 278 97.93 98.04 -0.11 latvian 28000 640 88.67 87.31 1.56 polish 52000 991 95.99 95.12 0.91 portuguese 176000 375 98.2 98.26 -0.06 romanian 157000 451 97.11 97.19 -0.08 russian 58400 715 96.0 95.07 0.98 slovak 64800 1186 93.25 92.43 0.89 slovenian 96500 1101 97.07 96.9 0.17 turkish 31200 972 95.81 95.01 0.85 urdu 101000 1001 96.76 97.12 -0.37 Table 1: Here we present the number of tokens in each of the UD treebanks we use as well as the number of morphological tags. For the case of Latvian, we find that the operation { replace : s  a} is the most common error made by our model. This operation corresponds to a possible issue in the Latvian treebank, where adjectives were marked with gendered lemmas. This issue has now been resolved in the latest version of the treebank. For Estonian, the operation { insert : m, insert : a} is the most common error. The suffix -ma in Estonian is used to indicate the infinitive form of verbs. Gold lemmata for verbs in Estonian are marked in their infinitive forms whereas our system predicts the stems of these verbs instead. These inflected forms are usually ambiguous and we believe that the model doesn't generalize well to different form-lemma pairs, partly due to fewer training data available for Estonian. This is an example of an error pattern that could be corrected using improved morphological information about the tokens. Finally, in Arabic, we find that the most common error pattern corresponds to a single ambiguous word form, 'an , which can be lemmatized as 'anna (like that in English) or 'an (like to in English) depending on the usage of the word in A r a b i c B a s qu e C r oa t i a n D u t c h E s t o n i a n F i nn i s h G er m a n G ree k H i nd i H un ga r i a n I t a li a n L a t v i a n P o li s h P o r t u g u e s e R o m a n i a n R u ss i a n S l ova k S l ov e n i a n T u r k i s h U r du Ambiguous Unseen Seen Unambiguous All 93.395.4 (10.6k) 92.692.0(2.1k) 90.188.2(1.5k) 93.395.3(0.9k) 87.386.3(1.0k) 93.292.3(1.6k) 98.898.7(1.7k) 96.595.8(1.5k) 97.597.5(7.4k) 98.398.2(1.5k) 96.396.3(2.1k) 78.576.8(0.6k) 95.892.7(0.8k) 97.597.0(2.6k) 95.095.4(2.0k) 88.678.4(0.3k) 95.595.5(0.9k) 94.895.3(1.3k) 84.785.2(0.3k) 95.496.0(7.0k) 60.761.9(2.4k) 91.588.7(4.1k) 86.684.4(1.8k) 92.491.4(1.5k) 69.165.4(3.6k) 86.584.0(4.2k) 93.085.7(2.0k) 83.677.1(2.0k) 90.691.9(1.4k) 92.088.9(3.6k) 91.088.8(0.8k) 77.073.5(2.8k) 89.086.9(2.5k) 94.893.3(1.0k) 87.986.3(1.8k) 90.587.9(3.1k) 85.582.9(4.2k) 91.390.1(2.6k) 90.586.6(2.6k) 91.292.4(0.8k) 98.597.6 (13.7k) 99.499.1 (13.4k) 99.299.0(9.1k) 98.899.0(7.5k) 97.397.2(5.3k) 99.198.8(9.3k) 99.299.3(6.8k) 99.699.2(5.5k) 99.899.8 (23.8k) 98.998.6(4.4k) 99.399.3(7.4k) 98.097.1(4.3k) 99.499.1(5.3k) 99.399.5(5.6k) 99.299.1 (10.5k) 99.599.2(5.8k) 99.399.1(5.2k) 99.399.2(8.2k) 99.399.1(5.3k) 99.299.2(5.9k) 93.193.6 (26.6k) 96.796.6 (19.6k) 96.295.7 (12.5k) 97.397.6(9.8k) 85.885.0(9.9k) 94.894.3 (15.1k) 97.597.7 (10.5k) 95.394.2(8.9k) 98.998.9 (32.6k) 96.195.0(9.5k) 97.998.0 (10.3k) 88.787.3(7.7k) 96.095.1(8.5k) 98.298.3(9.2k) 97.197.2 (14.2k) 96.095.1(9.1k) 93.292.4 (10.3k) 97.196.9 (12.1k) 95.895.0(8.2k) 96.897.1 (13.7k) -10% -5% 0% 5% 10% Figure 5: Dev accuracy breakdown by type of inflected form on all languages comparing our system with greedy decoding against our run of Lematus-ch20, colored by relative improvement in percentage. context. The word 'anna must be followed by a nominal sentence while 'an is followed by a verb. Hence, models that can incorporate rich contextual information would be able to avoid such errors. Simply presenting improved results does not entirely satiate our curiosity: we would also like to understand why our model performs better. Specifically, we have assumed an additional level of supervisionnamely, the annotation of morphological tags. We provide the differences between our method and our retraining of the Lematus system presented in Table 1. In addition to the perfor-Pearson's Rv Spearman's  # tags vs.  0.206 0.209 # tokens vs.  -0.808 -0.845 Table 2: The table shows the correlations between the differences in dev performance between our model with greedy decoding and Lematus and two aspects of the data: number of tokens and number of tags. mance of the systems, we also list the number of tokens in each treebank and the number of distinct morphological tags per language. We perform a correlational study, which is shown in Table 2. Morphological Complexity and Performance. We see that there is a moderate positive correlation (  = 0 . 209 ) between the number of morphological tags in a language and the improvement our model obtains. As we take the number of tags as a proxy for the morphological complexity in the language, we view this as an indication that attempting to directly extract the relevant morpho-syntactic information from the corpus is not as effective when there is more to learn. In such languages, we recommend exploiting the additional annotation to achieve better results. Amount of Data and Performance. The second correlation we find is a stronger negative correlation (  =  0 . 845 ) between the number of tokens available for training in the treebank and the gains in performance of our model over the baseline. This is further demonstrated by the learning curve plot in Figure 6, where we plot the validation accuracy on the Polish treebank for different sizes of the training set. The gap between the performance of our model and Lematus-ch20 is larger when fewer training data are available, especially for ambiguous tokens. This indicates that the incorporation of morphological tags into a model helps more in the low-resource setting. Indeed, this conclusion makes senseneural networks are good at extracting features from text when there is a suf-ficiently large amount of data. However, in the low-resource case, we would expect direct supervision on the sort of features we desire to extract to work better. Thus, our second recommendation is to model tags jointly with lemmata when fewer training tokens are available. As we noted earlier, it is almost always the case that token-level annotation of lemmata comes with token-level annotation of morphological tags. In low-resource scenarios, a data augmentation approach such as the one proposed by Bergmanis and Goldwater (2019) can be helpful and serve complementary to our approach. We have presented a simple joint model for morphological tagging and lemmatization and discussed techniques for training and decoding. Empirically, we have shown that our model achieves state-of-the-art results, hinting that explicitly modeling morphological tags is a more effective manner for modeling context. In addition to strong numbers, we tried to explain when and why our model does better. Specifically, we show a significant correlation between our scores and the number of tokens and tags present in a treebank. We take this to indicate that our method improves performance more for low-resource languages as well as morphologically rich languages. We thank Toms Bergmanis for his detailed feedback on the accepted version of the manuscript. Additionally, we would like to thank the three anonymous reviewers for their valuable suggestions. The last author would like to acknowledge support from a Facebook Fellowship.
abstain abstain abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain objective result objective abstain objective method method abstain abstain other abstain method other other other method other other other other abstain other other other other other abstain method other abstain abstain abstain abstain abstain method method abstain method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain method method method abstain abstain method abstain method abstain abstain abstain method abstain method abstain method method other other method method other method method method method other other other other method method other other other other other other abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result abstain result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain result result abstain result result abstain abstain abstain abstain result abstain other result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain result result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain method result abstain result result other other other