Aspect-based sentiment classification is a popular task aimed at identifying the corresponding emotion of a specific aspect. One sentence may contain various sentiments for different aspects. Many sophisticated methods such as attention mechanism and Convolutional Neural Networks (CNN) have been widely employed for handling this challenge. Recently, semantic dependency tree implemented by Graph Convolutional Networks (GCN) is introduced to describe the inner connection between aspects and the associated emotion words. But the improvement is limited due to the noise and instability of dependency trees. To this end, we propose a dependency graph enhanced dual-transformer network (named DGEDT) by jointly considering the flat representations learnt from Transformer and graph-based representations learnt from the corresponding dependency graph in an iterative interaction manner. Specifically, a dual-transformer structure is devised in DGEDT to support mutual reinforcement between the flat representation learning and graph-based representation learning. The idea is to allow the dependency graph to guide the representation learning of the transformer encoder and vice versa. The results on five datasets demonstrate that the proposed DGEDT outperforms all state-of-the-art alternatives with a large margin. Aspect-based or aspect-level sentiment classification is a popular task with the purpose of identifying the sentiment polarity of the given aspect (Yang et al., 2017; Zhang and Liu, 2017; Zeng et al., 2019). The goal is to predict the sentiment polarity of a given pair (sentence, aspect). Aspects in our study are mostly noun phrases appearing in the  Corresponding author. input sentence. As shown in Figure 1, where the comment is about the laptop review, the sentiment polarities of two aspects battery life and memory are positive and negative, respectively. Giving a specific aspect is crucial for sentiment classification owing to the situation that one sentence sometimes contains several aspects, and these aspects may have different sentiment polarities. Modern neural methods such as Recurrent Neural Networks (RNN), Convolutional Neural Networks (CNN) (Dong et al., 2014; Vo and Zhang, 2015) have already been widely applied to aspect-based sentiment classification. Inspired by the work (Tang et al., 2016a) which demonstrates the importance of modeling the semantic connection between contextual words and aspects, RNN augmented by attention mechanism (Bahdanau et al., 2015; Luong et al., 2015; Xu et al., 2015) is widely utilized in recent methods for exploring the potentially relevant words with respect to the given aspect (Yang et al., 2017; Zhang and Liu, 2017; Zeng et al., 2019; Wang et al., 2016). CNN based attention methods (Xue and Li, 2018; Li et al., 2018) are also proposed to enhance the phrase-level representation and achieved encouraging results. Although attention-based models have achieved promising performance on several tasks, the limitation is still obvious because attention module may highlight the irrelevant words owing to the syntactical absence. For example, given the sentence it has a bad memory but a great battery life. and aspect battery life , attention module may still assign a large weight to word bad rather than great , which adversely leads to a wrong sentiment polarity prediction. To take advantages of syntactical information among aspects and contextual words, Zhang et al. (2019) proposed a novel aspect-based GCN method which incorporates dependency tree into the attention models. Actually, using GCN (Kipf and Aspect: memory Sentiment: Negative Aspect: battery life Sentiment: Positive Figure1:Atypicalutterancesampleofaspect-basedsentimentclassificationtaskwithaproperdependencytree,noticethatdifferentaspectsmayhavedifferentsentimentpolarities. Welling,2017)toencodetheinformationconveyedbyadependencytreehasalreadybeeninvestigatedinseveralfields, e.g., modelingdocument-wordre- (2018) used GCN over dependency trees in document dating and relation classification, respectively. Yao et al. (2019) introduced GCN to text classification task with the guidance of document-word and word-word relations. Furthermore, Zhang et al. (2019) introduced aspect-based GCN to cope with aspect-level sentiment classification task using dependency graphs. On the other hand, Chen and Qian (2019) introduced and adapted Capsule Networks along with transfer learning to improve the performance of aspect-level sentiment classification. Gao et al. (2019) introduced BERT into a target-based method, and Sun et al. (2019) constructed BERT-based auxiliary sentences to further improve the performance. Since Transformer (Vaswani et al., 2017) and GCN are two crucial sub-modules in DGEDT, here we briefly introduce these two networks and illustrate the fact that GCN can be considered as a specialized Transformer. Assume that there are three input matrices Q  R n  d k , K  R m  d k , V  R m  d v , which represent the queries, keys and values respectively. n and m are the length of two inputs. Q (cid:48) = Attention ( Q, K, V ) = softmax ( QKT  d k ) V, (1) where Q (cid:48)  R n  d v , d k and d v are the dimension size of keys and values, respectively. Actually, Transformer adopts multi-head attention mechanism to further enhance the representative ability as follows: h i = Attention ( QW Qi , KW Ki , V W Vi ) , (2) Q (cid:48) = Concat ([ h 1 , ... ]) WO , (3) where i  [1 , H ] , H is the head size, W Qi  R d k  d k /H , W Ki  R d k  d k /H , W Vi  R d v  d v /H and WO  R d v  d v , and h i is the i -th head embedding. Then, two normalization layers are employed to extract higher-level features as follows: Q (cid:48) 1 = Norm ( Q (cid:48) + Q ) , (4) Q (cid:48) 2 = Norm ( Q (cid:48) 1 + F F N ( Q (cid:48) 1 )) , (5) where F F N ( x ) = Relu ( xW 1 + b 1 ) W 2 + b 2 is a two-layer multi-layer perceptron (MLP) with the activation function Relu , Norm is a normalization Dual-transformer Structure Max-Pooling Classify Attention Module Dependency Graph (Aspect-modified) Aspect Representation Aspect Span Aspect Span SUMSUM Aspect Representation BiLSTM/BERT Input Figure 2: An overall demonstration of our proposed DGEDT. layer, Q (cid:48) 2 is the output vector of this transformer layer. Equations (1)-(5) can be repeated for T times. Note that if Q = K = V , this operation can be considered as self alignment. As for GCN, the computation can be conducted as follows when the adjacent matrix of each word in the input is explicitly provided. Q (cid:48) = Norm ( Q + Relu ( 1 | A adj | A adj QW )) , (6) where A adj  R n  n is the adjacent matrix formed from the dependency graph, n is the number of words, Q  R n  d k , W  R d k  d k . 1 | A adj | A adj is similar to softmax ( QKT  d k ) which is denoted as a generated alignment matrix, except for the main difference that A adj is fixed and discrete. It is obvious that Equation (6) can be decomposed into Equations (1)-(4), and it can be also repeated for T times. In our perspective, GCN is a specialized Transformer with the head size set to one and the generated alignment matrix replaced by a fixed adjacent matrix. The network architecture of our proposed DGEDT is shown in Figure 2. For a given input text, we Input Embedding Self Attention Feed Forward Add&Norm Add&Norm BiGCN Add&Norm Flat (with graph) Graph (with flat) MutualBiaffine Add&Norm Add&Norm T  Figure 3: A simplified demonstration of dualtransformer structure, which consists of two submodules, one is a standard transformer, another is a transformer-like structure implemented by BiGCN withthesupervisionofdependencygraph. the representations of two directions to produce the final output in each iteration, while other similar methods conduct the merging only in the last iteration. BiGCN represents Equations (9)-(11). We use a simple method to merge the adjacent matrix of the words in the same aspect span as follows: A (cid:48) adj i = MIN ( (cid:126) 1 , SUM ([ A adj spani ])) , (13) where A adj can be replaced by A outadj and A inadj , and we can thus get A outadj (cid:48) and A inadj (cid:48) . Each span records the start and end position of the given aspect. span i denotes the i-th span in original text. BiAffine Module: Assume that there are two inputs S 1  R n  h and S 2  R n (cid:48)  h , we introduce a mutual BiAffine transformation process to interchange their relevant features as follows: A 1 = softmax ( S 1 W 1 ST 2 ) , (14) A 2 = softmax ( S 2 W 2 ST 1 ) , (15) S (cid:48) 1 = A 1 S 2 , (16) S (cid:48) 2 = A 2 S 1 , (17) S (cid:48) 1 , S (cid:48) 2 = Biaffine ( S 1 , S 2 ) , (18) where W 1 , W 2  R h  h . Here, S (cid:48) 1 can be considered as a projection from S 2 to S 1 , and S (cid:48) 2 follows the same principle. Biaffine represents Equations (14)-(17). A 1 and A 2 are temporary alignment matrices projecting from S 2 to S 1 and S 1 to S 2 , respectively. The Whole Procedure: We can then assemble all the sub-modules mentioned above to construct our proposed dual-transformer structure, and the detailed procedures are listed below: S Tr (cid:48) t = T ransfomer ( S Trt ) , (19) SG (cid:48) t = BiGCN ( S Gt , A outadj (cid:48) , A inadj (cid:48) ) , (20) S Tr (cid:48)(cid:48) t , SG (cid:48)(cid:48) t = Biaffine ( S Tr (cid:48) t , SG (cid:48) t ) , (21) S Trt +1 = Norm ( S Tr (cid:48) t + S Tr (cid:48)(cid:48) t ) , (22) S Gt +1 = Norm ( SG (cid:48) t + SG (cid:48)(cid:48) t ) , (23) where S Tr 0 = SG 0 = H , and H  RN s  h denotes the contextual hidden representations { s 1 , ... } from the aspect-based encoder. T ransfomer represents the process denoted by Equations (1)-(5). Equations (19)-(23) can be repeatedly calculated for T times and t  [0 , T ] . We choose S TrT (flat (with graph) in Figure 3) as the last representation, because SGT (graph (with flat) in Figure 3) heavily depends on the dependency graph. Given M aspect representations can be obtained through the above mentioned procedure, we can derive the final aspect representation by Max-Pooling operation. Here, we utilize an attention mechanism to identify relevant words with respect to the aspect. However, these would be M aspect representations which are all highly relevant to the aggregated aspect representation. To avoid that these aspect mentions from being assigned with too high weight, we utilize a mask mechanism to explicitly set the attention values of aspect mentions to zeros. Let I be the index set of these M aspect mentions, we form Mask vector as follows: Mask i = (cid:40)  inf, if i  I ; 0 , if other. h f = MaxP ooling ([ S TrTi | i  I ]) , (25) a f = softmax ( h f W 3 S TrTT + Mask ) , (26) h (cid:48) f = Relu ([ h f , a f S TrT ] W (cid:48) + b (cid:48) ) , (27) p = softmax ( h (cid:48) f W p + b p ) , (28) where W 3 , W (cid:48) , W p and b (cid:48) , b p are learnable weights and biases, respectively. The proposed DGEDT is optimized by the standard gradient descent algorithm with the cross-entropy loss and L2-regularization: where D denotes the training dataset, y p is the ground-truth label and p y p means the y p -th element of p . represents all trainable parameters, and  is the coefficient of the regularization term. Our experiments are conducted on five datasets, including one (Twitter) which is originally built by Dong et al. (2014), and the other four datasets (Lap14, Rest 14, Rest 15, Rest16) are respectively from SemEval 2014 task 4 (Pontiki et al., 2014), SemEval 2015 task 12 (Pontiki et al., 2015) and SemEval 2016 task 5 (Hercig et al., 2016), consisting We compare the proposed DGEDT  with a line of baselines and state-of-the-art alternatives, including LSTM, MemNet (Tang et al., 2016b), AOA (Huang et al., 2018), IAN (Ma et al., 2017), TNet-LF (Li et al., 2018), CAPSNet (Chen and Qian, 2019), Transfer-CAPS (Chen and Qian, 2019), TG-BERT (Gao et al., 2019), AS-CNN (Zhang et al., 2019) and AS-GCN (Zhang et al., 2019). We conduct the experiments with our proposed DGEDT with BiLSTM as the aspect-based encoder, and DGEDT +BERT with BERT as the aspect-based encoder. Several simplified variants of DGEDT are also investigated: DGEDT(Transformer) denotes that we keep standard Transformer and remove the BiGCN part, DGEDT(BiGCN) denotes that we keep BiGCN and remove the Transformer part. The layer number or iteration number ( i.e., T ) of all available models is set to three for both Transformer and GCN. We use Spacy toolkit  to generate dependency trees. We use BERT-base English version (Devlin et al., 2019), which contains 12 hidden layers and 768 hidden units for each layer. We use Adam (Kingma and Ba, 2014) as the optimizer for BERT and our model with the learning rate initialized by 0.00001 and 0.001 respectively, and decay rate of learning is set as 0.98. Except for the influence of decay rate, the learning rate decreases dynamically according to the current step number. Batch shuffling  available at https://github.com/tomsonsgs/DGEDT-senti-master. is applied to the training set. The hidden size of our basic BiLSTM is 256 and the size of all embeddings is set as 100. The vocab size of BERT is 30,522. The batch size of all model is set as 32. As for regularization, dropout function is applied to word embeddings and the dropout rate is set as 0.3. Besides, the coefficient  for the L2-norm regularization is set as 0.0001. We train our model up to 50 epochs and conduct the same experiment for 10 times with random initialization. Accuracy and Macro-Averaged F1 are adopted as the evaluation metrics. We follow the experimental setup in (Zhang et al., 2019; Chen and Qian, 2019) and report the average maximum value for all metrics on testing set. If the model is not equipped with BERT, then we use word vectors that were pre-trained from Glove (Pennington et al., 2014). As shown in Table 2, our model DGEDT outperforms all other alternatives on all five dataset. BERT makes further improvement on the performance especially in Twitter, Rest14 and Rest 15. We can conclude that traditional Transformer DGEDT(Transformer) obtains better performance than DGEDT(BiGCN) in the most datasets. DGEDT employs and combines two sub-modules (traditional Transformer and dependency graph enhanced GCN) and outperforms any single submodule. Using dependency tree indeed contributes to the performance when acting as a supplement rather than a single decisive module. Note that the performance of individual modules is already reported in Table 2. As shown in Table 3, we investigate and report four typical ablation conditions. Mask' denotes that we remove the aspect-based attention mask mechanism, and MultiAspect' denotes that we only use the aspect representation of the first aspect mention instead of MaxPooling them. We can see that these two procedures provide slight improvement. BiGCN(+GCN)' means that we remove the bidirectional connection and only use original GCN, the results show that bidirectional GCN outperforms original GCN owing to the adequate connection information. BiAffine' indicates that we remove the BiAffine process and use all the outputs of dual-transformer structure, we can thus conclude that BiAffine process is critical for our model, and utilizing simple concatenation of the Model Twitter Lap14 Rest14 Rest15 Rest16 Acc F1 Acc F1 Acc F1 Acc F1 Acc F1 LSTM 69.6 67.7 69.3 63.1 78.1 67.5 77.4 55.2 86.8 63.9 MemNet 71.5 69.9 70.6 65.2 79.6 69.6 77.3 58.3 85.4 66.0 AOA 72.3 70.2 72.6 67.5 80.0 70.4 78.2 57.0 87.5 66.2 IAN 72.5 70.8 72.1 67.4 79.3 70.1 78.6 52.7 84.7 55.2 TNet 73.0 71.4 74.6 70.1 80.4 71.0 78.5 59.5 89.1 70.4 AS-CNN 71.1 69.5 72.6 66.7 81.7 73.1 78.5 58.9 87.4 64.6 CAPSNet   72.7 68.8 78.8 69.7     Transfer-CAPS   73.9 70.2 79.3 70.9     AS-GCN 72.2 70.4 75.6 71.1 80.8 72.0 79.9 61.9 89.0 67.5 DGEDT(Transformer) 74.1 72.7 76.0 71.4 82.8 73.9 81.0 64.9 90.0 72.6 DGEDT(BiGCN) 72.8 71.0 76.2 71.8 81.8 72.5 80.4 62.9 89.4 70.4 DGEDT 74.8 73.4 76.8 72.3 83.9 75.1 82.1 65.9 90.8 73.8 TG-BERT 76.7 74.3 78.9 74.4 85.1 78.4     DGEDT-BERT 77.9 75.4 79.8 75.6 86.3 80.0 84.0 71.0 91.9 79.0 Table 2: Overall performance of accuracy and F1 on five datasets, AS means aspect-based. outputs of Transformer and BiGCN is worse than DGEDT(Transformer). As shown in Figure 4, we find that three is the best iteration number for Lap14 and Rest14. Dependency information will not be fully broadcasted when the iteration number is too small. The model will suffer from over-fitting and redundant information passing, which results in the performance drop when iteration number is too large. So, numerous experiments need to be conducted to figure out a proper iteration number. As shown in Figure 5, DGEDT and DGEDT(BiGCN) output correct prediction Negative while DGEDT(Transformer) fails for the sentence The management was less than accommodating . To figure out the essential cause, we demonstrate the attention of self alignment in Figure 5. We can see that for the aspect management , DGEDT(Transformer) mainly focuses on accommodating , which is a positive word at document level. Thus, DGEDT(Transformer) obtains an incorrect prediction Positive . In the dependency tree, less which is often regarded as a negative word has a more related connection with aspect management , so DGEDT(BiGCN) outputs right sentiment Negative . With the assistance of supplementary dependency graph, DGEDT also obtains right prediction Negative owing to the high attention value between management and less . As shown in Figure 6, DGEDT and DGEDT(Transformer) output correct prediction Positive while DGEDT(BiGCN) fails for the sentence This little place is wonderfully warm welcoming . To figure out the essential cause, we demonstrate the attention of self alignment and dependency tree in Figure 6. We can see that for the aspect place , DGEDT(Transformer) mainly focuses on wonderfully , which is a positive word at document level. Thus, DGEDT(Transformer) obtains a correct prediction Positive . In the dependency tree, little which is often regarded as a negative word has a more related connection with aspect place , so DGEDT(BiGCN) outputs incorrect sentiment Negative . With the disturbance of inappropriate dependency tree, DGEDT still Aspect: management Golden: Negative DGEDT(Transformer): Positive DGEDT(BiGCN): Negative DGEDT: Negative obtains right prediction Positive owing to the high attention value between place and wonderfully . We can see from two examples above that DGEDT is capable of achieving the proper bal-ance between dependency graph enhanced BiGCN and traditional Transformer according to different situations. Recently neural structures with syntactical information such as semantic dependency tree and constituent tree are widely employed to enhance the word-level representation of traditional neural networks. These structures are often modeled and described by TreeLSTMs or GCNs. To introduce Transformer into our task and diminish the error induced by incorrect dependency trees, we propose a dual-transformer structure which considers the connections in dependency tree as a supplementary GCN module and a Transformer-like structure for self alignment in traditional Transformer. The results on five datasets demonstrate that dependency tree indeed promotes the final performance when utilized as a sub-module for dual-transformer structure. In future work, we can further improve our method in the following aspects. First, the edge information of the dependency trees needs to be exploited in later work. We plan to employ an edge-aware graph neural network considering the edge labels. Second and last, domain-specific knowledge can be incorporated into our method as an external learning source. We thank the reviewers for their valuable comments. This work is supported through the grants from National Natural Science Foundation of China (NSFC-61772378), the National Key research and Development Program of China (No.2017YFC1200500) and the Major Projects of the National Social Science Foundation of China (No.11&ZD189).
abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain method abstain abstain abstain other abstain abstain method abstain method abstain method method other abstain abstain abstain abstain other method method method abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain result result other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result abstain method abstain other other