For over a decade, machine learning has been used to extract opinion-holder-target structures from text to answer the question Who expressed what kind of sentiment towards what? . Recent neural approaches do not outperform the state-of-the-art feature-based models for Opinion Role Labeling (ORL). We suspect this is due to the scarcity of labeled training data and address this issue using different multi-task learning (MTL) techniques with a related task which has substantially more data, i.e. Semantic Role Labeling (SRL). We show that two MTL models improve significantly over the single-task model for labeling of both holders and targets, on the development and the test sets. We found that the vanilla MTL model which makes predictions using only shared ORL and SRL features, performs the best. With deeper analysis we determine what works and what might be done to make further improvements for ORL. Fine-Grained Opinion Analysis (FGOA) aims to: (i) detect opinion expressions (O) that convey attitudes such as sentiments, agreements, beliefs or intentions (like feared in example (1)), (ii) measure their intensity (e.g. strong), (iii) identify their holders (H), i.e. entities that express an attitude (e.g. it ), (iv) identify their targets (T), i.e. entities or propositions at which the attitude is directed (e.g. violence ) and (v) classify their target-dependent attitude (e.g. negative sentiment) 1 . 1 Examples are drawn from MPQA (Wiebe et al., 2005). holders and targets), the task is usually approached with sequence labeling techniques and the BIO encoding scheme (Choi et al., 2006; Yang and Cardie, 2013; Katiyar and Cardie, 2016). Initially pipeline models were proposed which first predict opinion expressions and then, given an opinion, label its opinion roles , i.e. holders and targets (Kim and Hovy, 2006; Johansson and Mos-chitti, 2013). Pipeline models have been substituted with so-called joint models that simultaneously identify all opinion entities, and predict which opinion role is related to which opinion (Choi et al., 2006; Yang and Cardie, 2013; Katiyar and Cardie, 2016). Recently an LSTM-based joint model was proposed (Katiyar and Cardie, 2016) that unlike the prior work (Choi et al., 2006; Yang and Cardie, 2013) does not depend on external resources (such as syntactic parsers or named entity recognizers). The neural variant does not outperform the feature-based CRF model (Yang and Cardie, 2013) in Opinion Role Labeling (ORL). Both the neural and the CRF joint models achieve about 55% F1 score for predicting which targets relate to which opinions in MPQA. Thus, these models are not yet ready to answer the question this line of research is usually motivated with: Who expressed what kind of sentiment towards what? . Our goal is to investigate the limitations of neural models in solving different subtasks of FGOA on MPQA and to gain a better understanding of what is solved and what is next. We suspect that one of the fundamental obstacles for neural models trained on MPQA is its small size. One way to address scarcity of labeled data is to use multi-task learning (MTL) with appropriate auxiliary tasks. A promising auxiliary task candidate for ORL is Semantic Role Labeling (SRL), the task of predicting predicate-argument structure of a sentence, which answers the question Who did what to whom, where and when? . Table 1 illustrates the output of the SRL demo 2 for example (1), following the PropBank SRL scheme (Palmer et al., 2005) 3 . SRL4ORL. The semantic roles of the predicate fear (marked blue bold) correspond to the opinion roles H and T, according to MPQA. For this reason, the output of SRL systems has been commonly used for feature-based FGOA models (Kim and Hovy, 2006; Johansson and Moschitti, 2013; Choi et al., 2006; Yang and Cardie, 2013). Additionally, a considerable amount of training data is available for training SRL models (Table 2 in Sec. 3), which made neural SRL models successful (Zhou and Xu, 2015; Yang and Mitchell, 2017). Obstacles. Although SRL is similar in nature to ORL, it cannot solve ORL for all cases (Rup-penhofer et al., 2008). In example (2) holder and target of the predicate please correspond to A1, A0 semantic roles respectively, wheres for the predicate fear in (1) holder and target correspond to A0, A1 respectively. We took into account this observation when deciding on an appropriate MTL model by splitting its parameters into shared and task-specific ones (i.e. hard-parameter sharing). (2) [I] A 1 H am very [ pleased ] O pos that [the Council has now approved the Kyoto Protocol thus enabling the EU to proceed with its ratification] A 0 T . A further obstacle for properly exploiting SRL training data with MTL could be specificities, inconsistency and incompleteness of the MPQA annotations. In example (3), Rice expressed his negative sentiment towards the three countries in question by setting the criteria which states something negative about those countries: they are repressive and grave human rights violators [...] . In this case, the model should not pick any local semantic role for the target. In examples (45), the same opinion expression concerned realizes different scopes for the target. A model which exploits SRL knowledge could be biased to always label targets as complete SRL role constituents, as in example (5). (4) Rice told us [the administration] H was [ concerned ] O neg that [Iraq] T would take advantage of the 9/11 attacks. (5) [The Chinese government] H is deeply [ concerned ] O neg about [the sudden deterioration in the Middle East situation] T , Tang said. Regarding incompleteness, prior work (Kati-yar and Cardie, 2016) has shown that their model makes reasonable predictions in sentences which do not have annotations at all, e.g. [mothers] H [ care ] O for [their young] T , in: From the fact that mothers care for their young, we can not deduce that they ought to do so, Hume argued. The examples above show that incorporating SRL knowledge via multi-task learning is a reasonable way to improve ORL, but at the same time they alert us that given the specificities of MPQA and ORL annotations in general, it is not obvious whether MTL can overcome divergences in the annotation schemes of opinion and semantic role labeling. We investigate this research question by adopting one of the recent successful architectures for SRL (Zhou and Xu, 2015) and experiment with different multi-task learning frameworks. Our contributions are: (i) we adapt a recently proposed neural SRL model for ORL, (ii) we enhance the model using different MTL techniques with SRL to tackle the problem of scarcity of labeled data for ORL, (iii) we show that most of the MTL models improve the single-task model for labeling of both holders and targets on development and test sets, and two of them make yield significant improvements, (iv) by deeper analysis we provide a better understanding of what is solved and where to head next for neural ORL. Neural multi-task learning (MTL) receives a lot of attention and new MTL architectures emerge regularly. Yet there is no clear consensus which MTL architecture to use in which conditions. We experiment with well-received architectures that could adapt to different cases of ORL from Section 1. As a general neural architecture for singleand multi-task learning we use the recently proposed SRL model (Zhou and Xu, 2015) (Z&X-STL) which successfully labels semantic roles without any syntactic guidance. This model consists of a stack of bi-directional LSTMs and a CRF which makes the final prediction. The inputs to the first LSTM are not only token embeddings but three additional features: embedding of the predicate, embedding of the context of the predicate and an indicator feature (1 if the current token is in the predicate context, 0 otherwise). Thus, every sentence is processed as many times as there are predicates in it. Adapting this model for labeling of opinion roles is straightforward, the only difference being that opinion expressions can be multiwords and only two opinion roles are assigned. MTL techniques aim to learn several tasks jointly by leveraging knowledge from all tasks. In the context of neural networks, MTL is commonly used in such a way that it is predefined which layers have tied parameters and which are task-specific (i.e. hard-parameter sharing). There are various ways of defining which parameters should be shared and how to train them. Fully-shared (FS) MTL model. A fully-shared model (Fig. 1) shares all parameters of the general model except the output layer. Each task has a task-specific output layer which makes the prediction based on the representation produced by the final LSTM. When training on a mini-batch of a certain task, parameters of the output layer of the Figure 3: (Adversarial) state-private ((A)SP) MTL. other tasks are not updated. This model should be effective for constructions with a clear mapping between opinion and semantic roles such as { H 7 A0, T 7 A1 } as in example (1) (Sec. 1). Hierarchical MTL (H-MTL) model. For NLP applications, often some given (high-level) task is supposed to benefit from another (low-level) task more than the other way around, e.g. parsing from POS tagging. This intuition lead to designing hierarchical MTL models (Sgaard and Goldberg, 2016; Hashimoto et al., 2017) in which predictions for low-level tasks are not made on the basis of the representation produced at the final LSTM, but on the representation produced by a lower-layer LSTM (Fig. 2). Task-specific layers atop shared layers could potentially give the model more power to distinguish or ignore certain semantic roles. If so, this MTL model is more suitable for examples like (2) and (3) (Sec. 1). Shared-private (SP) MTL model. In the state-private model, in addition to the stack of shared LSTMs, each task has a stack of task-specific LSTMs (Liu et al., 2017) (Fig. 3). Representations at the outermost shared LSTM and the task-specific LSTM are concatenated and passed to the task-specific output layer. The ORL representation produced independently from SRL gives the model the ability to utilize the shared and entirely task-specific information. For labeling of targets, it is expected that for examples (1) & (5) the model relies mostly on the shared representation, for examples (2) & (4) on both shared and ORL-specific representations, and for example (3) solely on the ORL-specific representation. Adversarial shared-private (ASP) model. The limitation of the SP model is that it does not prevent the shared layers from capturing task-specific features. To ensure this, ASP extends the 585 task train size dev size test size |Y| CoNLL'05 SRL 90750 3248 6071 106 MPQA (4-CV) ORL 3141.25 1055 1036.75 7 MPQA (10-CV) ORL 3516.3 1326 349.3 7 Table 2: Datasets w/ nb. SP model with a task discriminator (Liu et al., 2017). The task discriminator (Fig. 3, marked red) predicts to which task the current batch of data belongs, based on the representation produced by the shared LSTMs. If the shared LSTMs are task-invariant, the discriminator should perform badly. Thus, we update the shared parameters to maximize the discriminator's cross-entropy loss. At the same time we want the discriminator to challenge the shared LSTMs, so we update the discriminator's parameters to minimize its cross-entropy loss. This minmax optimization is known as adversarial training and recently it gained a lot of attention for NLP applications (Liu et al., 2017; Chen et al., 2017; Kim et al., 2017; Qin et al., 2017; Wu et al., 2017; Gui et al., 2017; Li et al., 2017; Zhang et al., 2017; Joty et al., 2017). For SRL we use the newswire CoNLL-2005 shared task dataset (Carreras and M`arquez, 2005), annotated with PropBank predicate-argument structures. Sections 2-21 of the WSJ corpus (Char-niak et al., 2000) are used for training and section 24 as dev set. The test set consists of section 23 of WSJ and 3 sections of the Brown corpus. For ORL we use the manually annotated MPQA 2.0. corpus (Wiebe et al., 2005; Wilson, 2008). It mostly contains news documents, but also travel guides, transcripts of spoken conversations, e-mails, fundraising letters, textbook chapters and translations of Arabic source texts. For both tasks we adopt evaluation metrics from prior work. For SRL, precision is defined as the proportion of semantic roles predicted by a system 4 Examples how to use our scripts can be found at https://github.com/amarasovic/ naacl-mpqa-srl4orl/blob/master/generate_mpqa_jsons.py . which are correct, recall is the proportion of gold roles which are predicted by a system, F1 score is the harmonic mean of precision and recall. In case of ORL, we report 10-fold CV 5 and repeated 4-fold CV with binary F1 score and proportional F1 score , for holders and targets separately. Binary precision is defined as the proportion of predicted holders (targets) that overlap with the gold holder (target), binary recall is the proportion of gold holders (targets) for which the model predicts an overlapping holder (target). Proportional recall measures the proportion of the overlap between a gold holder (target) and an overlapping predicted holder (target), proportional precision measures the proportion of the overlap between a predicted holder (target) and an overlapping gold holder (target). F1 scores are the harmonic means of the corresponding precision and recall. We evaluate our models using two evaluation settings. First, we follow Katiyar and Cardie (2016) which set aside 132 documents for development and used the remaining 350 documents for 10-fold CV. However, in the 10-fold CV setting, the size of the tests sets is 3 times smaller than the dev set size (Table 2, row 3), and, consequently, results in high-variance estimates on the test sets. Therefore we additionally evaluate our models with 4-fold CV. We set aside 100 documents for development and use 25% of the remaining documents for testing. The resulting test sets are comparable in size to the dev set (Table 2, row 2). We run 4-fold CV twice with two different random seeds. We do not tune hyperparameters (HPs), but follow suggestions proposed in the thorough HP study for sequence labeling tasks (Reimers and Gurevych, 2017). HPs can be found in the Supplementary Material. We evaluate all models after every (cid:6) train size batch size (cid:7) iteration on the ORL dev set and save them if they achieve a higher arithmetic mean of proportional F1 scores of holders and targets on the ORL dev set. The saved models are used for testing. We report the mean of F1 scores over 10 folds and the standard deviation (appears as a subscript) of all models in Table 3. We report the mean of F1 scores over 4 folds and 2 different seed (8 evalua-5 We used the same splits as the prior work (Katiyar and Cardie, 2016). We thank the authors for providing the splits. We mark significant difference between MTL models and the single-task (Z&X-STL) model, observed using a Kolmogorov-Smirnov signifi-cance test ( p < 0 . 05 ) (Massey Jr, 1951), with  in superscript and between the FS-MTL model and other MTL models with 3 . STL vs. MTL. In the 10-fold CV evaluation setting (Table 3), the FS-MTL and the H-MTL models improve over the Z&X-STL model in all evaluation measures, for both holders and targets. When evaluated in the repeated 4-fold CV setting (Table 4), all MTL models improve over the Z&X-STL model in all evaluation measures, for both holders and targets. The FS-MTL and the H-MTL models improve significantly in all evaluation measures, for both holders and targets, on both dev and test sets, when evaluated with repeated 4-fold CV. With 10-fold CV the improvements are also significant, except for targets on the test set. This is probably due to the small size of the test sets (Table 2, row 3), which results in a high-variance estimate. Indeed, standard deviations on the 10-fold CV test sets are always much higher compared to the dev set or to the test sets of 4-fold CV. It is not surprising that larger improvements are visible in the labeling of holders. They are usually short, less ambiguous and often presented with the A0 semantic role, whereas annotating targets is a challenging task even for humans. 6 Larger improvements are visible for proportional F1 score than for binary F1 score. That is, more data and SRL knowledge helps the model to better annotate the scope of opinion roles. Comparing MTL models. In Section 2 we introduced MTL models with task-specific LSTM layers hypothesizing that these layers should give MTL models more power to adapt to a variety of potentially problematic cases that we illustrated in the Introduction. However, our results show that the FS-MTL model performs significantly better or comparable to MTL models that include task-specific layers. Reimers and Gurevych (2017) show that MTL is especially sensitive to the selection of HPs. Thus, a firm and solid comparison of the different MTL models requires thorough HP optimization, to properly control the number of parameters and regularization of the models. We leave HP optimization for future work. Our aim in this section is to analyze what the proposed models are good at, in which ways MTL improves over the single-task ORL model and what could be done to achieve further progress. models on the ORL dev set using 4-fold CV repeated twice with different seeds (8 evaluation tri-als). We say that a model predicts a role of a given opinion expression correctly if the model predicts a role that overlaps with the correct role in at least 6 out of 8 evaluation trials. If a model predicts a role that overlaps with the correct role in at most 2 out of 8 trials, we say that the model predicts the role incorrectly . The requirement on 6-8 (in)correct predictions reduces the risk of analyzing inconsistent predictions and enables us to draw firmer conclusions. We analyze the following scenarios: (i) both the FS-MTL model and the Z&X-STL model make correct predictions (Tables 56) (ii) the FS-MTL model makes a correct prediction, while the Z&X-STL makes an incorrect prediction (Tables 1112) (iii) both models make wrong predictions (Tables 910) In the following, we categorize predictions in case as hard cases . In Tables 56 and 912, the opinion expression is bolded, the correct role is italicized, predictions of the FS-MTL model are colored blue (subscript FS ), predictions of the Z&X-STL model are colored yellow (subscript ZX ) and green marks predictions where both models agree. For simplicity, we show only holders or targets, although the models predict both roles jointly. What works well? There are 668/1055 instances in the dev set for which both models predict holders correctly, and 663/1055 for targets. Examples 15 in Table 5 suggest that holders that can be properly labeled by both models ( easy 588 1 It would be entirely improper if , in its defense of Israel FS , the United States continues to exert pressure on [...] . cases) are subjects of their governing heads or A0 roles. The statistics in Table 7 (col. 1, rows 2 3) supports this observation. 7 In contrast, holders that both models predict incorrectly ( hard cases) are less frequently subjects or A0 roles (col. 2, rows 23). Also, easy holders are close to the corresponding opinion expression: the average distance is 1 . 54 tokens (Table 7, row 4), contrary to the hard holders with the average distance of 7 . 56 . Examples 15 in Table 6 suggest that targets that can be properly labeled by both models are objects of their governing heads or A1 roles. Table 8, row 3, shows that the majority of the easy targets are indeed A1 roles, in contrast to the hard targets. Similar to holders, the easy targets are in average 7 tokens closer to the opinion expression. What to do for further improvement? There are 165/1055 instances in the dev set for which 7 The statistics is calculated using the output of mate-tools (Bjorkelund et al., 2010). both models predict holders incorrectly, and 176 for targets. As we have seen so far, many holders that are subjects or A0 roles, and targets that are A1 roles, are properly labeled by both models. However, a considerable amount of such holders and targets are not correctly predicted (Table 78, col. 2, rows 23). Thus our models do not work flawlessly for all such cases. A distinguishing property of the hard cases is the distance of the role from the opinion. Thus, future work should advance the model' s ability to capture long-range dependencies. Examples in Table 9 demonstrate that holders, harder to label with our models, occur with the corresponding opinions in more complicated syntactic constructions. In the first example, the FS-MTL model does not recognize the possessive and is possibly biased towards picking the country ( Is-real ), which occurs immediately after the opinion. In the second example, the opinion expression is 589 1 Yoshihisa Murasawa , a management consultant for Booz-Allen & Hamilton Japan Inc. Z&X-STL model predicts incorrectly in 6/8 trials. a nominal predicate and the holder is its object. The sentence is in passive voice but the models probably interpret it in the active voice and thus make the wrong prediction. In the third example, the opinion expression is the head of the relative clause that modifies the holder. These examples raise the following questions: would improved consistency with syntax lead to improvements for ORL and could we train a dependency parsing model with SRL and ORL to help the models handle syntactically harder cases? Example 4 shows that holders specific to the MPQA annotation schema are hard to label as they require inference skills: from the department said , we can defeasibly infer that it is the department who expects [this cost] to stand at just $ 400/year [...] . To handle such cases, it would be worth trying training our models jointly with models for recognizing textual entailment. Examples 67 illustrate that some gap in performance stems from difficulties in processing MPQA. Example 5 has no gold holder, but the models make plausible predictions. For example 6, FS-MTL predicts the discontinuous holder they ... all , while MPQA allows only contiguous entities. Therefor our evaluation scripts interpret they and all as two separate holders and deem all as incorrect, resulting with lower precision. Finally, for example 7 our models make plausible predictions. However, the gold holder is always the entity from the coreference cluster that is the closest to the opinion. 8 The evaluation scripts needs to be extended such that predicting any entity from the coreference cluster is considered to be correct. To conclude, to better evaluate future developments, it would be worth curating MPQA instances with missing roles and extending evaluation to account for coreferent holders and discontinuous roles. dif-8 We followed the prior work (Katiyar and Cardie, 2016). ficulties in labeling targets originate from similar reasons as for holders. Examples 13 demonstrate complex syntactic constructions, examples 46 MPQA-specific annotations that require inference and example 7 exemplifies a missing target. How does MTL help? There are 18/1055 instances in the dev set for which the FS model predicts the holder correctly and the Z&X-STL model does not, and 19/1055 for targets. For holders, for 9 out 18 of such examples, the Z&X-STL model does not predict anything (as in examples 25 in Table 11). From examples 1 5 we notice that SRL data helps to handle more complex syntactic constructions. From examples 57 we observed that using MTL with SRL helps to handle cases when more than one person or organization is present in the close neighborhood of the opinion. For targets, for 11 out of 18 cases the Z&X-STL model does not predict anything as in examples 12 in Table 11. We conclude that the greatest improvements from the FS-MTL model comes from having far fewer missing roles. FGOA. Closest to our work are Yang and Cardie (2013) (Y&C) and Katiyar and Cardie (2016) (K&C). They as well label both holders and targets in MPQA. By contrast, our focus is on the task of ORL. We thus refrain from predicting opinion expressions first, to ensure a reproducible evaluation setup on a fixed set of gold opinion expressions. The MTL models we develop in this work will, however, be the basis for the full task in a later stage. Because of these differences, direct comparison to Y&C and K&C is not possible. However, if we compare our results we notice a big gap that demonstrates that opinion expression extraction is the import step in FGOA. Similar to K&C, Liu et al. (2015) jointly labels opinion expressions and their targets in reviews. Some work focuses entirely on labeling of opinion expressions (Yang and Cardie, 2014; Irsoy and Cardie, 2014). Other work looks into specific subcategories of ORL: opinion role induction for verbal predicates (Wiegand and Ruppenhofer, 2015), categorization of opinion words into actor and speaker view (Wiegand et al., 2016b), opinion roles extraction on opinion compounds (Wiegand et al., 2016a). Wiegand and Ruppenhofer (2015) report 72 . 54 binary F1 score for labeling of holders in MPQA (results for targets are not reported). Neural SRL. New neural SRL models have emerged (He et al., 2017; Yang and Mitchell, 2017; Marcheggiani and Titov, 2017) since we started this work. In future work we can improve our models with such new proposals. Auxiliary tasks for MTL. Other work investigates under which conditions MTL is effective. Martnez Alonso and Plank (2017) show that the best auxiliary tasks have low kurtosis of labels (usually a small label set) and high entropy (la-bels occur uniformly). We show that the best MTL model for ORL is the model which uses shared layers only. Thus it seems reasonable to consider only a small and uniform SRL label set { A0, A1 } . Bingel and Sgaard (2017) show that MTL works when the main task has a flattening learning curve, but the auxiliary task curve is still steep. We notice such behavior in our learning curves. We address the problem of scarcity of annotated training data for labeling of opinion holders and targets (ORL) using multi-task learning (MTL) with Semantic Role Labeling (SRL). We adapted a recently proposed neural SRL model for ORL and enhanced it with different MTL techniques. Two MTL models achieve significant improvements with all evaluation measures, for both holders and targets, on both dev and test set, when evaluated with repeated 4-fold CV. We recommend evaluation with comparable dev and test set sizes for future work, as this enables more reliable evaluation. With deeper analysis we show that future developments should improve the ability of the models to capture long-range dependencies, investigate if consistency with syntax can improve ORL, and consider other auxiliary tasks such as dependency parsing or recognizing textual entailment. We emphasize that future improvements can be measured more reliably if opinion expressions with missing roles are curated and if the evaluation considers all mentions in opinion role coreference chains as well as discontinuous roles. This work has been supported by the German Research Foundation as part of the Research Training Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) under grant No. GRK 1994/1.
abstain abstain abstain method result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain abstain abstain abstain method objective result result abstain objective objective method result objective abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method other method abstain abstain method abstain abstain method other abstain method abstain abstain abstain method method abstain method method abstain method method abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain other objective objective objective other objective other other other other other other abstain objective other other other abstain other other abstain method objective abstain method objective objective other other