We introduce a new syntax-aware model for dependency-based semantic role labeling that outperforms syntax-agnostic models for English and Spanish. We use a BiLSTM to tag the text with supertags extracted from dependency parses, and we feed these supertags, along with words and parts of speech, into a deep highway BiLSTM for semantic role labeling. Our model combines the strengths of earlier models that performed SRL on the basis of a full dependency parse with more recent models that use no syntactic information at all. Our local and non-ensemble model achieves state-of-the-art performance on the CoNLL 09 English and Spanish datasets. SRL models benefit from syntactic information, and we show that supertagging is a simple, powerful, and robust way to incorporate syntax into a neural SRL system. Semantic role labeling (SRL) is the task of identifying the semantic relationships between each predicate in a sentence and its arguments (Gildea and Jurafsky, 2002). While early research assumed that SRL models required syntactic information to perform well (Punyakanok et al., 2008), recent work has demonstrated that neural networks can achieve competitive and even state-of-the-art performance without any syntactic information at all (Zhou and Xu, 2015; Marcheggiani et al., 2017; He et al., 2017). These systems have the benefits of being simpler to implement and performing more robustly on foreign languages and out-of-domain data, cases where syntactic parsing is more difficult (Marcheggiani et al., 2017). In this paper, we show that using supertags is an effective middle ground between using full syntactic parses and using no syntactic information  Work partially done at Yale University. at all. A supertag is a linguistically rich description assigned to a lexical item. Supertags impose complex constraints on their local context, so supertagging can be thought of as almost parsing (Bangalore and Joshi, 1999). Supertagging has been shown to facilitate Tree-Adjoining Grammar (TAG) parsing (Bangalore et al., 2009; Friedman et al., 2017; Kasai et al., 2017, 2018) and Combinatory Categorial Grammar (CCG) parsing (Clark and Curran, 2007; Kummerfeld et al., 2010; Lewis et al., 2016; Xu, 2016). We propose that supertags can serve as a rich source of syntactic information for downstream tasks without the need for full syntactic parsing. Following Ouchi et al. (2014), who used supertags to improve dependency parsing, we extract various forms of supertags from the dependency-annotated CoNNL 09 corpus. This contrasts with prior SRL work that uses TAG or CCG supertags (Chen and Rambow, 2003; Lewis et al., 2015). We train a bidirectional LSTM (BiLSTM) to predict supertags and feed the predicted supertag embedding, along with word and predicted part-of-speech embeddings, to another BiLSTM for semantic role labeling. Predicted supertags are represented by real-valued vectors, contrasting with approaches based on syntactic paths (Roth and Lapata, 2016; He et al., 2018) and syntactic edges (Marcheggiani and Titov, 2017; Strubell et al., 2018). This way of incorporating information alleviates the issue of error propagation from parsing. Supertagging has many advantages as part of a natural language processing pipeline. First, as a straightforward sequence-labeling task, the supertagging architecture is much simpler than comparable systems for structured parsing. Second, it is simple to extract different forms of supertags from a dependency corpus to test different hypotheses about which kinds of syntactic information are most useful for downstream tasks. Our re-Token Model 1 Model TAG No DEP/R DEP/R , P/R P/R it SBJ/R was ROOT+L R ROOT+SBJ/L PRD/R n't ADV/L ADV/L black NAME/R NAME/R Monday PRD/L+L Table 1: Supertags for the sentence No, it wasn't black Monday. sults show that supertags, by encoding just enough information, can improve SRL performance even compared to systems that incorporate complete dependency parses. We experiment with four supertag models, two from Ouchi et al. (2014), one from Nguyen and Nguyen (2016), and one of our own design inspired by Tree Adjoining Grammar supertags (Bangalore and Joshi, 1999). Each model encodes a different set of attributes about the syntactic relationship between a word, its parent, and its dependents. Table 2 summarizes what information is expressed in each supertag model. Model 0. A Model 0 supertag for a word w encodes the dependency relation and the relative position (direction) between w and its head, i.e. left (L), right (R), or no direction (ROOT) (Nguyen and Nguyen, 2016). Model 1. A Model 1 supertag for w adds to the parent information from Model 0 the information of whether w possesses dependents to its left (L) or right (R) (Ouchi et al., 2014). Model 2. A Model 2 supertag for w extends Model 1 by encoding the dependency relation between w and its obligatory dependents. 1 When w 1 Following Ouchi et al. (2014), we define obligatory dependents as those with relations SBJ,' OBJ,' PRD,' and VC.' For Spanish, we define obligatory syntactic arguments lacks such obligatory children, we encode whether it possesses non-obligatory dependents to the left (L) or right (R) as in Model 1. Model TAG. We propose Model TAG supertags that represent syntactic information analogously to TAG supertags (elementary trees) (Bangalore and Joshi, 1999). A Model TAG supertag encodes the dependency relation and the direction of the head of a word similarly to Model 0 if the dependency relation is non-obligatory (corresponding to adjunction nodes), and the information about obligatory dependents of verbs if any similarly to Model 2 (corresponding to substitution nodes). Motivated by recent state-of-the-art supertaggers (TAG: Kasai et al. (2017, 2018); CCG: Lewis et al. (2016); Xu (2016)), we employ a bi-directional LSTM (BiLSTM) architecture for our supertagging. The input for each word is the conncate-nation of a dense vector representation of the word, a vector embedding of a predicted PTB-style POS tag (only for English), 2 and a vector output by character-level Convolutional Neural Networks (CNNs) for morphological information. For POS tagging before English supertagging, we use the same hyperparameters as in Ma and Hovy (2016). For supertagging, we follow the hyperparameters chosen in Kasai et al. (2018) regardless of the supertag model that is employed. We initialize the word embeddings by the pretrained 100 dimensional GloVe (Pennington et al., 2014) and the 300 dimensional FastText (Bo-janowski et al., 2017) vectors for English and Spanish respectively. Our SRL model is most similar to the syntax-agnostic SRL model proposed by Marcheggiani et al. (2017). Our model differs in two ways: 1) we add randomly initialized 50 dimensional supertag embeddings to the input layer (Fig. 1), and 2) we use a modified LSTM with highway layers and regularization (0.5 dropout) as in He et al. (2017). We use the same hyperparameters as in Marcheggiani et al. (2017) with randomly initialized 50 dimensional embeddings for supertags. 3 as dc,'suj,' cd,' and cpred.' 2 For the English data, predicted PTB-style POS tags generally contribute to increases, approximately 0.2-0.4% in the dev set, whereas for Spanish adding predicted (coarse-grained) POS tags hurt the performance. 3 We provide lists of hyperparameters in Appedix A.1. For pre-trained word embeddings, we use the same word embeddings as the ones in Marcheggiani et al. (2017) for English and the 300-dimensional FastText vectors (Bojanowski et al., 2017) for Spanish. We use the predicates predicted by the mate-tools (Bjorkelund et al., 2009) (English) and Zhao et al. (2009) (Spanish) system in our models, again following Marcheggiani et al. (2017) to facilitate comparison. Our code is available online for easy replication of our results. 4 Figure 1: SRL architecture with a highway BiLSTM. Table 3 provides our supertagging results for English and Spanish across the different types of supertag described above. Here we clearly see the general pattern that the more granular supertagging becomes, the less reliable it is, and finding the balance between granularity and predictability is critical. We present our SRL results in Tables 4-7 along with the results from a baseline 4 https://github.com/jungokasai/ stagging_srl . BiLSTM model, which is our implementation of the syntax-agnostic model in Marcheggiani et al. (2017). We also present results for a BiLSTM model with dropout and highway connections but without supertags (BDH model), to distinguish the effects of supertags from the effects of better LSTM regularization. In every experiment we train the model five times, and present the mean score. Table 4 shows that Model 1 yields the best performance in the English dev set, and thus we only use Model 1 supertags for test evaluation. We primarily show results only with word type embeddings to conduct fair comparisons with prior work, but we also provide results with deep contextual word representations, ELMo (Peters et al., 2018), and compare our results with recent work that utilizes ELMo (He et al., 2018). 5 English in-domain. Table 5 summarizes the results on the English in-domain test set. First, we were able to approximately replicate the results from Marcheggiani et al. (2017). Adding dropout and highway connections to our BiLSTM model improves performance by 0.5 points, to 88.1, and adding supertags improves results even further to 88.6. Our supertag model performs even better than the non-ensemble model in Marcheggiani and Titov (2017), in which the model is given the complete dependency parse of the sentence. This result suggests that supertags can be even more effective for SRL than a more complete representation of syntax. Furthermore, our supertag-based method with contextual representations achieves 90.2, a new state-of-the-art. Interestingly, the gain from supertagging decreases to 0.2 points (90.2 vs. 90.0) in the presence of contextual representations, suggesting that contextual representations encode some of the same syntactic information that supertags provide. English out-of-domain. One of the advantages of using a syntax-agnostic SRL model is that such a model can perform relatively well on out-of-domain data, where the increased difficulty of syn-5 We used the pretrained ELMo available at https:// tfhub.dev/google/elmo/2 . tactic parsing can cause errors in a syntax-based system (Marcheggiani et al., 2017). Unfortunately we were not able to replicate the out-of-domain results of Marcheggiani et al. (2017): our implementation of the BiLSTM achieves a score of 76.4, compared to their reported score of 77.7. However, we note that incorporating supertags into our own model improves performance, with our best model achieving a score of 77.6. Our supertag-based model also substantially outperforms the full dependency-based models (Roth and Lapata, 2016; Marcheggiani and Titov, 2017). This suggests that syntax with a certain degree of granularity is useful even across domains. Our supertag-based method alleviates the issue of error propagation from syntactic parsing. Finally, our model with contextual representations yields 80.8, an improvement of 1.5 F1 points over the previous state-of-the-art (He et al., 2018), which also uses ELMo. Spanish. Table 7 shows the results on the Spanish test data. Our BiLSTM implementation yields lower performance than Marcheggiani et al. (2017): our model achieves a score of 79.1, compared to their reported score of 80.3. However, our BDH model yields a score of 80.8, already achieving state-of-the-art performance. Adding supertags to BDH improves the score further to 81.0. This suggests that while the gains are relatively small, the supertag-based approach still helps Spanish SRL. Supertags slightly improve performance when contextual representations are used (83.0 vs. 82.9). See appendices for details. Following the analysis in Roth and Lapata (2016), we show plots of the BiLSTM, BDH (BiLSTM + Dropout + Highway), and Model 1 role labeling performance for sentences with varying number of words (in-domain: Fig. 2; out-of-domain: Fig. 3). Note first that BDH outperforms the baseline BiLSTM model in a relatively uniform manner across varying sentence lengths. The benefits of Model 1 supertags, in contrast, come more from longer sentences, especially in the out-Non-ensemble System P R F 1 FitzGerald et al. (2015)   87.3 Roth and Lapata (2016) 90.0 85.5 87.7 Marcheggiani et al. (2017) 88.7 86.8 87.7 Marcheggiani and Titov (2017) 89.1 86.8 88.0 BiLSTM 88.5 86.7 87.6 BDH 88.3 87.8 88.1 BDH + Model 1 89.0 88.2 88.6 + Contextual Representations He et al. (2018) (ELMo) 89.7 89.3 89.5 BDH + ELMo 90.3 89.7 90.0 BDH + Model 1 + ELMo 90.3 90.0 90.2 Ensemble System FitzGerald et al. (2015)   87.7 Roth and Lapata (2016) 90.3 85.7 87.9 Marcheggiani and Titov (2017) 90.5 87.7 89.1 Table 5: Results on the CoNLL 2009 in-domain test set for English. of-domain test set. This implies that the supertag model is robust to the sentence length, probably because supertags encode relations between words that are linearly distant in the sentence, information that a simple BiLSTM is unlikely to recover. Table 8 reports SRL results broken down by predicate category (V: Verb, Propbank; N: Noun, Nombank) and semantic role. We can observe that the various supertag models differ in their performance for different predicate-role pairs, suggesting that different kinds of linguistic information are relevant for identifying the different roles. Overall, Model 1 supertags achieve the most consistent improvements over BiLSTM and BiLSTM + Dropout + Highway (BDH) in V / A0, V / A1, V / A2, V / AM, N / A2, and N / AM. Moreover, Model 1 even improves on Path-LSTM (Roth and Lapata, 2016) by large margins in V / A0, V / A1, V / AM, and N / AM, even though the Path-LSTM model has the benefit of using the complete dependency path between each word and its head. This shows that supertags can be even more effective for SRL than more granular syntactic informationeven quite simple supertags, like Model 0, which encode only the dependency arc between a word and its head. We presented state-of-the-art SRL systems on the CoNLL 2009 English and Spanish data that make crucial use of dependency-based supertags. We showed that supertagging serves as an effective middle ground between syntax-agnostic approaches and full parse-based approaches for dependency-based semantic role labeling. Supertags give useful syntactic information for SRL and allow us to build an SRL system that does not depend on a complex architecture. We have also seen that the choice of the linguistic content of a supertag makes a significant difference in its utility for SRL. In this work, all models are developed independently for English and Spanish. However, sharing some part of SRL models could improve performance (Mulcaire et al., 2018, 2019). In future work, we will explore crosslingual transfer for supertagging and semantic role labeling. The authors thank Diego Marcheggiani for assistance in implementing SRL models and Diego Marcheggiani and the anonymous reviewers for their helpful feedback. This work was funded in part by the Funai Overseas Scholarship to JK.
objective method method result result abstain abstain abstain result abstain abstain abstain abstain objective result abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method abstain abstain method abstain method method method abstain abstain result result other abstain result abstain result result abstain abstain abstain result abstain abstain abstain abstain abstain other abstain result result result abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain method result method result objective abstain objective other other