What logic rules can be learned using ELMo?
1).But   2).Eng  3). A-But-B
We analyze the performance of different sentiment classification models on syntactically complex inputs like A-but-B sentences. The first contribution of this analysis addresses reproducible research: to meaningfully compare different models, their accuracies must be averaged over far more random seeds than what has traditionally been reported. With proper averaging in place, we notice that the distillation model described in arXiv:1603.06318v4 [cs.LG], which incorporates explicit logic rules for sentiment classification, is ineffective. In contrast, using contextualized ELMo embeddings (arXiv:1802.05365v2 [cs.CL]) instead of logic rules yields significantly better performance. Additionally, we provide analysis and visualizations that demonstrate ELMo's ability to implicitly learn logic rules. Finally, a crowdsourced analysis reveals how ELMo outperforms baseline models even on sentences with ambiguous sentiment labels.
In this paper, we explore the effectiveness of methods designed to improve sentiment classification (positive vs. negative) of sentences that contain complex syntactic structures. While simple bag-of-words or lexicon-based methods BIBREF1 , BIBREF2 , BIBREF3 achieve good performance on this task, they are unequipped to deal with syntactic structures that affect sentiment, such as contrastive conjunctions (i.e., sentences of the form “A-but-B”) or negations. Neural models that explicitly encode word order BIBREF4 , syntax BIBREF5 , BIBREF6 and semantic features BIBREF7 have been proposed with the aim of improving performance on these more complicated sentences. Recently, hu2016harnessing incorporate logical rules into a neural model and show that these rules increase the model's accuracy on sentences containing contrastive conjunctions, while PetersELMo2018 demonstrate increased overall accuracy on sentiment analysis by initializing a model with representations from a language model trained on millions of sentences. In this work, we carry out an in-depth study of the effectiveness of the techniques in hu2016harnessing and PetersELMo2018 for sentiment classification of complex sentences. Part of our contribution is to identify an important gap in the methodology used in hu2016harnessing for performance measurement, which is addressed by averaging the experiments over several executions. With the averaging in place, we obtain three key findings: (1) the improvements in hu2016harnessing can almost entirely be attributed to just one of their two proposed mechanisms and are also less pronounced than previously reported; (2) contextualized word embeddings BIBREF0 incorporate the “A-but-B” rules more effectively without explicitly programming for them; and (3) an analysis using crowdsourcing reveals a bigger picture where the errors in the automated systems have a striking correlation with the inherent sentiment-ambiguity in the data.