Critical to natural language generation is the production of correctly inflected text. In this paper, we isolate the task of predicting a fully inflected sentence from its partially lemmatized version. Unlike traditional morphological inflection or surface realization, our task input does not provide gold tags that specify what morphological features to realize on each lemmatized word; rather, such features must be inferred from sentential context. We develop a neural hybrid graphical model that explicitly reconstructs morphological features before predicting the inflected forms, and compare this to a system that directly predicts the inflected forms without relying on any morphological annotation. We experiment on several typologically diverse languages from the Universal Dependencies treebanks, showing the utility of incorporating linguistically-motivated latent variables into NLP models. NLP systems are often required to generate grammatical text, e.g., in machine translation, summarization, dialogue, and grammar correction. One component of grammaticality is the use of contextually appropriate closed-class morphemes. In this work, we study contextual inflection , which has been recently introduced in the CoNLL-SIGMORPHON 2018 shared task (Cotterell et al., 2018) to directly investigate context-dependent morphology in NLP. There, a system must inflect partially lemmatized tokens in sentential context. For example, in English, the system must reconstruct the correct word sequence two cats are sitting from partially lemmatized sequence two _cat_ are sitting . Among other things, this requires: (1) identifying cat as a noun in this context, (2) recognizing that cat should be inflected as plural to agree with the nearby verb and numeral, and (3) realizing this inflection as the suffix s . Most past work in supervised computational morphologyincluding the previous CoNLL-SIGMORPHON shared tasks on morphological reinflection (Cotterell et al., 2017)has focused mainly on step (3) above. As the task has been introduced into the literature only recently, we provide some background. Contextual inflection amounts to a highly constrained version of language modeling. Language modeling predicts all words of a sentence from scratch, so the usual training and evaluation metric perplexityis dominated by the language model's ability to predict content , which is where most of the uncertainty lies. Our task focuses on just the ability to reconstruct certain missing parts of the sentenceinflectional morphemes and their orthographic realization. This refocuses the modeling effort from semantic coherence to morphosyntactic coherence, an aspect of language that may take a back seat in current language models (see Linzen et al., 2016; Belinkov et al., 2017). Contextual inflection does not perfectly separate grammaticality modeling from content modeling: as illustrated in Tab. 1, mapping two cats _be_ sitting to the fully-inflected two cats were sitting does not require full knowledge of English grammarthe system does not have to predict the required word order nor the required auxiliary verb be , as these are supplied in the input. Conversely, this example does still require predicting some contentthe semantic choice of past tense is not given by the input and must be guessed by the system. 1 The primary contribution of this paper is a novel structured neural model for contextual inflection. The model first predicts the sequence of morphological tags from the partially lemmatized sequence and, then, it uses the predicted tag and lemma to inflect the word. We use this model 1 This morphological feature is inherent in the sense of Booij (1996). to evince a simple point: models are better off jointly predicting morphological tags from context than directly learning to inflect lemmata from sentential context. Indeed, none of the participants in the 2018 shared task jointly predicted tags with the inflected forms. Comparing our new model to several competing systems, we show our model has the best performance on the majority of languages. We take this as evidence that predicting morphological tags jointly with inflecting is a better method for this task. Furthermore, we provide an analysis discussing the role of morphological complexity in model performance. Given a language, let M be a set of morphological tags in accordance with the Universal Dependencies annotation (Nivre et al., 2016). Each m  M has the form m = (cid:104) t,  (cid:105) , where t is a part of speech, and the slot  is a set of attributevalue pairs that represent morphosyntactic information, such as number, case, tense, gender, person, and others. We take t  T , the set of universal parts of speech described by Petrov et al. (2012). A sentence consists of a finite word sequence w (we use boldface for sequence variables). For every word w i in the sequence, there is a corresponding analysis in terms of a morphological tag m i  M and a lemma (cid:96) i . In general, w i is determined by the pair (cid:104) (cid:96) i , m i (cid:105) . 2 Using this notation, Cotterell et al. (2018)'s shared task is to predict a sentence w from its partially lemmatized form (cid:96) , inferring m as an intermediate latent variable. Our dataset (3) has all three sequences for each sentence. Consider an extreme case when all words are lemmatized. 3 We introduce a structured neural model 2 Although w i can sometimes be computed by concatenating (cid:96) i with m i -specific affixes, it can also be irregular. 3 In case of partially lemmatized sequence we still train the model to predict the tags over the entire sequence, but evaluate it only for lemmatized slots. (1) In other words, the distribution is over interleaved sequences of one-to-one aligned inflected words and morphological tags, conditioned on a lemmatized sequence, all of length n . This distribution is drawn as a hybrid (directedundirected) graphical model (Koller and Friedman, 2009) in Fig. 1. We define the two conditional distributions in the model in 2.2 and 2.3, respectively. The distribution p ( m | (cid:96) ) is defined to be a conditional random field (CRF; Lafferty et al., 2001). In this work, our CRF is a conditional distribution over morphological taggings of an input sequence. We define this conditional distribution as p ( m | (cid:96) ) = 1 Z ( (cid:96) ) n (cid:89) i =1  ( m i , m i  1 , (cid:96) ) (2) where  (  ,  ,  )  0 is an arbitrary potential, Z ( (cid:96) ) normalizes the distribution, and m 0 is a distinguished start-of-sequence symbol. In this work, we opt for a recurrent neural potentialspecifically, we adopt a parameterization similar to the one given by Lample et al. (2016). Our potential  is computed as follows. First, the sequence (cid:96) is encoded into a sequence of word vectors using the strategy described by Ling et al. (2015): word vectors are passed to a bidirectional LSTM (Graves et al., 2005), where the corresponding hidden states are concatenated at each time step. We simply refer to the hidden state h i  R d as the result of said concatenation at the i -th step. Using h i , we can define the potential function as  ( m i , m i  1 ) = exp (cid:0) A m i ,m i  1 + o (cid:62) m i h i (cid:1) , where A m i ,m i  1 is a transition weight matrix and o m i  R d is a morphological tag embedding; both are learned. The conditional distribution p ( w i | (cid:96) i , m i ) is parameterized by a neural encoderdecoder model with hard attention from Aharoni and Goldberg (2017). The model was one of the top performers in the 2016 SIGMORPHON shared task (Cotterell et al., 2016); it achieved particularly high accuracy in the low-resource setting. Hard attention is motivated by the observation that alignment between the input and output sequences is often monotonic in inflection tasks. In the model, the input lemma is treated as a sequence of characters, and encoded using a bidirectional LSTM (Graves and Schmidhu-ber, 2005), to produce vectors x j for each character position j . Next the word w i = c = c 1    c | w i | is generated in a decoder character-by-character: p ( c j | c <j ,l i , m i ) = (3) softmax ( W   ( z 1 , . . . , z j ) + b ) where z j is the concatenation of the current attended input x j alongside morphological features, m i , and an embedding of the previously generated symbol c j  1 ; and finally  is an LSTM over the sequence of z j vectors. The decoder additionally predicts a type of operation. 4 The distribution in Eq. (3), strung together with the other conditionals, yields a joint distribution over the entire character sequence: p ( c | (cid:96) i , m i ) = | w i | (cid:89) j =1 p ( c j | c <j , (cid:96) i , m i ) (4) 4 The model can be viewed as a transition system trained over aligned character-level strings to learn sequences of operations ( write or step ). For instance, to map the lemma talk to its past form talked , we feed in POS=V;Tense=PAST <w> t a l k </w> and train the network to output <w> t a l k e d </w> , where we have augmented the orthographic character alphabet  with the featureattribute pairs that constitute the morphological tag m i . We optimize the log-likelihood of the training data with respect to all model parameters. As Eq. (1) is differentiable, this is achieved with standard gradient-based methods. For decoding we use a greedy strategy where we first decode the CRF, that is, we solve the problem m (cid:63) = argmax m log p ( m | (cid:96) ) , using the Viterbi (1967) algorithm. We then use this decoded m (cid:63) to generate forms from the inflector. Note that finding the one-best string under our neural inflector is intractable, and for this reason we use greedy search. Dataset. We use the Universal Dependencies v1.2 dataset (Nivre et al., 2016) for our experiments. We include all the languages with information on their lemmata and fine-grained grammar tag annotation that also have fasttext embeddings (Bojanowski et al., 2017), which are used for word embedding initialization. 5 Evaluation. We evaluate our model's ability to predict: (i) the correct morphological tags from the lemma context, and (ii) the correct inflected forms. As our evaluation metric, we report 1-best accuracy for both tags and word form prediction. Configuration. We use a word and character embedding dimensionality of 300 and 100, respectively. The hidden state dimensionality is set to 200. All models are trained with Adam (Kingma and Ba, 2014), with a learning rate of 0.001 for 20 epochs. Baselines. We use two baseline systems: (1) the CoNLLSIGMORPHON 2018 subtask 2 neural encoderdecoder with an attention mechanism (SM; Cotterell et al. (2018)), where the encoder represents a target form context as a concatenation of its lemma, its left and right word forms, their 5 We also choose mainly non-Wikipedia datasets to reduce any possible intersection with the data used for the FastText model training Language tag form JOINT GOLD JOINT DIRECTSM CPH Bulgarian 81 . lemmata and tag representations, and then the decoder generates the target inflected form character-by-character; and (2) a monolingual version of the best performing system of the shared task (CPH; Kementchedjhieva et al. (2018)) that augments the above encoderdecoder with full (sentence-level) left and right contexts (comprising of forms, their lemmata and morphological tags) as well as predicts morphological tags for a target form as an auxiliary task. 6 In both cases, the hyperparameters are set as described in Cotterell et al. (2018). We additionally evaluate the SIGMORPHON baseline system on prediction of the target form without any information on morphological tags ( DIRECT ). Tab. 2 presents the accuracy of our best model across all languages. 7 Below we highlight two main lessons from our error analysis that apply to a wider range of generation tasks, e.g., machine translation and dialog systems. Directly Predicting Morphology. Tab. 2 indicates that all systems that make use of morphological tags outperform the DIRECT baseline on most languages. The comparison of our hybrid model with latent morphological tags to the direct form generation baseline in SM suggests that we should be including linguistically-motivated latent vari-6 It has been shown to improve the model's performance. 7 The accuracy numbers are on average higher than the ones achieved in terms of the CoNLLSIGMORPHON 2018 subtask 2 since we did not filter out tokens that are typically not inflected (such as articles or prepositions). ables into models of natural language generation. We observe in Tab. 2 that predicting the tag together with the form (joint) often improves performance. The most interesting comparison here is with the multi-task CPH method, which includes morphology into the model without joint modeling; our model achieves higher results on 7/10 languages. Morphological Complexity Matters. We observed that for languages with rich case systems, e.g., the Slavic languages (which exhibit a lot of fu-sion), the agglutinative Finno-Ugric languages, and Basque, performance is much worse. These languages present a broader decision space and often require inferring which morphological categories need to be in agreement in order to make an accurate prediction. This suggests that generation in languages with more morphological complexity will be a harder problem for neural models to solve. Indeed, this problem is under-explored, as the field of NLP tends to fixate on generating English text, e.g., in machine translation or dialogue system research. Error Analysis. We focused error analysis on prediction of agreement categories. Our analysis of adjectivenoun agreement category prediction suggests that our model is able to infer adjective number, gender, and case from its head noun. Verb gender, which appears only in the past tense of many Slavic languages, seems to be harder to predict. Given that the linear distance between the subject and the verb may be longer, we suspect the network struggles to learn longer-distance dependencies, consistent with the findings of Linzen et al. (2016). Overall, automatic inference of agreement categories is an interesting problem that deserves more attention, and we leave it for future work. We also observe that most uncertainty comes from morphological categories such as noun number, noun definiteness (which is expressed morphologically in Bulgarian), and verb tense, all of which are inherent (Booij, 1996) 8 and typically cannot be predicted from sentential context if they do not participate in agreement. 9 On the other hand, aspect, although being closely related to tense, is well-predicted since it is mainly expressed as a separate lexeme. But, in general, it is still problematic to make a prediction in languages where aspect is morphologically marked or highly mixed with 8 Such categories exist in most languages that exhibit some degree of morphological complexity. Unless there is no strong signal within a sentence such as yesterday , tomorrow , or ago as in the case of tense. We additionally compared 1-best and 10-best predictions for tags. Most mispredictions existing in 1-best lists are due to inherent categories mentioned above (that allow multiple plausible options that can fit the sentential context). Indeed, the problem is usually solved by allowing system to output 10-best lists. There, precision@10 is on average 8 points higher than precision@1. Finally, our analysis of case category prediction on nouns shows that more common cases such as the nominative, accusative, and genitive are predicted better, especially in languages with fixed word order. On the other hand, cases that appear less frequently and on shifting positions (such as the instrumental), as well as those not associated with specific prepositions, are less well predicted. In addition, we evaluated the model's performance when all forms are replaced by their corresponding lemmata (as in two cat be sit ). For freer word order languages such as Polish or Latin, we observe a substantial drop in performance because most information on inter-word relations and their roles (expressed by means of case system) is lost. The primary evaluation for most contemporary language and translation modeling research is perplexity, BLEU (Papineni et al., 2002), or METEOR (Banerjee and Lavie, 2005). Undoubtedly, such metrics are necessary for extrinsic evaluation and comparison. However, relatively few studies have focused on intrinsic evaluation of the model's mastery of grammaticality. Recently, Linzen et al. (2016) investigated the ability of an LSTM language model to capture sentential structure, by evaluating subjectverb agreement with respect to number, and showed that under strong supervision, the LSTM is able to approximate dependencies. Taking it from the other perspective, a truer measure of grammatical competence would be a task of mapping a meaning representation to text, where the meaning representation specifies all necessary semantic contentcontent lemmata, dependency relations, and inherent closed-class morphemes (semantic features such as noun number, noun definiteness, and verb tense)and the system is to realize this content according to the morphosyntactic conventions of a language, which means choosing word order, agreement morphemes, function words, and the surface forms of all words. Such tasks have been investigated to some extentgenerating text from tectogrammatical trees (Hajic et al., 2002; Ptcek and abokrtsk, 2006) or from an AMR graph (Song et al., 2017). Belz et al. (2011) organized a related surface realization shared task on mapping unordered and uninflected dependency trees to properly ordered inflected sentences. The generated sentences were afterwards assessed by human annotators, making the task less scalable and more time consuming. Although our task is not perfectly matched to grammaticality modeling, the upside is that it is a lightweight task that works directly on text. No meaning representation is required. Thus, training and test data in any language can be prepared simply by lemmatizing a naturally occurring corpus. Finally, as a morphological inflection task, the form generation task is closely related to previous SIGMORPHON shared tasks (Cotterell et al., 2016, 2017). There, most neural models achieve high accuracy on many languages at type-level prediction of the form from its lemma and slot. The current task is more challenging in that the model has to perform token-level form generation and inherently infer the slot from the contextual environment. Our findings are in line with those from the CoNLL-SIGMORPHON 2018 shared task (Cotterell et al., 2018) and provide extra evidence of the utility of morphosyntactic features. This work proposed a method for contextual inflection using a hybrid architecture. Evaluation over several diverse languages showed consistent improvements over state of the art. Our analysis demonstrated that the contextual inflection can be a highly challenging task, and the inclusion of morphological features prediction is an important element in such a system. We also highlighted two types of morphological categories, contextual and inherent, in which the former relies on agreement and the latter comes from a speaker's intention. We thank all anonymous reviewers for their comments. The first author would like to acknowledge the Google PhD fellowship. The second author would like to acknowledge a Facebook Fellowship.
abstain method method objective result abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain objective abstain method abstain abstain objective method method abstain abstain method method abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain method method method abstain abstain method abstain abstain abstain abstain other abstain abstain other abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain result abstain abstain abstain result result abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain abstain abstain method abstain other other other other other other other other abstain other other other other other method objective abstain objective result other other other