We propose a novel attention network for document annotation with user-generated tags. The network is designed according to the human reading and annotation behaviour. Usually, users try to digest the title and obtain a rough idea about the topic first, and then read the content of the document. Present research shows that the title metadata could largely affect the social annotation. To better utilise this information, we design a framework that separates the title from the content of a document and apply a title-guided attention mechanism over each sentence in the content. We also propose two semantic-based loss regularisers that enforce the output of the network to conform to label semantics, i.e. similarity and subsumption. We analyse each part of the proposed system with two real-world open datasets on publication and question annotation. The integrated approach, Joint Multi-label Attention Network (JMAN), significantly outperformed the Bidirectional Gated Recurrent Unit (Bi-GRU) by around 13%-26% and the Hierarchical Attention Network (HAN) by around 4%-12% on both datasets, with around 10%-30% reduction of training time. Social annotation, or tagging, is a popular functionality allowing users to assign keywords to online resources for better semantic search and recommendation (Vander Wal, 2007; Singer et al., 2014; Gedikli and Jannach, 2014). Common socially annotated textual resources include questions, papers, (micro-)blogs, product reviews, etc. In practice, however, only a limited number of resources is annotated with tags. Annotating a large number of documents requires much cognitive effort and can be time-consuming. This has driven research on document annotation based on existing tag sets (Belem et al., 2017; Nie et al., 2014). Recent studies formalise the automated social annotation task as a multi-label classification problem (Gibaja and Ventura, 2015) and apply deep learning approaches (Li et al., 2016; Huang et al., 2016; Hassan et al., 2018). A strong baseline is the use of Bi-directional RNN (Schuster and Paliwal, 1997) with GRU (Cho et al., 2014) or LSTM (Hochreiter and Schmidhuber, 1997). Another more recent improvement is achieved through Hierarchical Attention Network (HAN) (Yang et al., 2016) which discriminates important words and sentences from others, as adapted in (Hassan et al., 2018) for annotation. These models, however, suffer from two issues: (i) simply scanning over the words and sentences, the models do not fully mimic the way users read and annotate documents, and (ii) semantic relations, similarity and subsumption, among the labels are not considered. Our model focuses on simulating users' reading and annotation behaviour with attention mechanisms. The title of a document is highly abstract while informative about the topics and has a direct impact on users' annotation choice (Lipczak and Milios, 2010), showing high descriptive capacity and effectiveness for annotation (Figueiredo et al., 2013); the content provides complementary information for annotation. Usually, users firstly read the title, and based on their understanding of the title, proceed to the content of the document. To simulate this behaviour, we propose an attention network with separated inputs (title and content) and parallelled attention layers at both the word-level and the sentence-level. One major distinction to previous approaches is to represent the content with a title-guided attention mechanism; this enables the network to discriminate among sentences based on its understanding of the title. various semantic forms and granularities (Peters, 2009; Heymann and Garcia-Molina, 2006). One challenging issue is how to exploit the relations among labels (user-generated tags) (Zhang and Zhou, 2014; Gibaja and Ventura, 2015) to improve the learning performance. Among neural network based methods, a recent attempt is to initialise weights for dedicated neurons in the last layer to memorise the label relations (Kurata et al., 2016; Baker and Korhonen, 2017), however, the limitation is the large number of neurons to be assigned, making it inefficient (or inapplicable) for systems with large number of labels. To incorporate the label semantics inferred from the data or from external knowledge bases into the network, we design two loss regularisers, for similarity and subsumption relations, respectively. The regularisers enforce the output layer of the network to satisfy the semantic constraints of the labels. We propose a parallelled two-layered attention network that simulates users' reading and annotation behaviour for document annotation. The proposed Joint Multi-label Attention Network (J-MAN) approach is depicted in Figure 1. The model inputs the title and content separately into two Bidirectional-RNNs with word-level attention and sentence-level attention mechanisms to capture the important words and sentences. Each target is a multi-hot (as opposed to an one-hot) representation of the labels in the label set y d  { 0 , 1 } | T | , where T is a list all labels, 1 indicates that a label appears in the label set of the document d , 0 otherwise. In Figure 1, attention mechanisms are indicated with dotted edges. One key distinction from the HAN model (Yang et al., 2016) is the title-guided sentence-level attention that models the reading order for annotation (the dotted edges linking c t and c ta ). The output layer s d =  ( W c c d + b c ) , activated with the sigmoid function  , is further constraint by two loss regularisers, emphasising two types of label relations, similarity and subsumption, respectively. For the RNN encoder, we apply the Gated Recurrent Unit (GRU) which can capture long ter-m dependencies and is usually more time-efficient than LSTM (Hochreiter and Schmidhuber, 1997) in training. The Bidirectional-GRU (Bi-GRU) encoder (Cho et al., 2014) concatenates the hidden states generated from two GRUs, one reading the Figure 1: The Proposed Joint Multi-label Attention Network (JMAN) for Social Text Annotation words (or sentences) forward and the other reading them backwards. This helps form a more complete understanding of the current word (or sentence). Hierarchical Attention captures the structure of a document by a word-level attention on each word's hidden state to create a sentence representation, then a sentence-level attention to form a content representation (Yang et al., 2016). The attention coefficients are computed based on the dot product between a non-linearly transformed weight vector of the hidden state and an informative vector, which encodes what is the most informative word (or sentence) in the sequence. This informative vector is commonly treated as a sequence of weights (Yang et al., 2016; Kumar et al., 2018; Hassan et al., 2018), trained along with other weights in the network. We applied parallelled word-level attention on the title and each sentence in the content. The attention coefficient and the final representation of a sequence is calculated as (taking words in title as an example): c t = (cid:88) i  i h i = (cid:88) i exp( v wt  v i ) (cid:80) j exp( v wt  v j ) h i (1) where v i = tanh( W t h i + b t ) is the output of a fully-connected layer of the hidden state h i for each word in the title, v wt is the informative vector for titles, and c t is the resulting title representation. We can compute each sentence representation c s and the content representation c a in a similar manner (see Figure 1). The attention mechanisms above do not capture the interaction between the title and content of the document. Title represents highly abstract while important information about the topics of a document. Selection of the important sentences in the content should conform to the document's general topic, e.g. title. We can thus model the title-guided sentence-level attention as: c ta = (cid:88) r  r h r = (cid:88) r exp( c t  v r ) (cid:80) k exp( c t  v k ) h r (2) where v r = tanh( W s h r + b s ) is a fully connected layer with the hidden state of the r th sentence h r as input and c t is the title representation obtained from Equation 1. Guiding sentence reading through title representation facilitates content understanding, but may lead to an overemphasis on the title in the annotation. In fact, the content itself, carrying more terms, conveys detailed information not covered by the title and may help suggest further tags for annotation (Figueiredo et al., 2013). We thus concatenate the title guided content representation c ta and the content representation c a from the original sentence-label attention, to form a more comprehensive representation of the content. The final content representation is then concatenated with the title representation c d = [ c t , c ta , c a ] . In the experiment, we will show the effectiveness of this design against several variations of the model. Users tend to annotate documents collectively with semantically related tags. Two major semantic relations in user-generated tags are similarity and subsumption (Stock, 2010; Peters, 2009). To deal with this label correlation issue, we propose two loss regularisers jointly learned with the binary cross entropy loss function. The intuition is that the output values of the neural network s d , having the dimensions as the label space | T | , should satisfy semantic relations among labels. Such relations can be inferred from the label sets or observed in external knowledge bases. The whole joint loss is defined as L = LCE +  1 L sim +  2 L sub . LCE is the binary cross entropy loss adopted for multilabel text classification (Nam et al., 2014). L sim and L sub are defined as: L sim = 1 2 (cid:88) d (cid:88) ( j,k ) | T j ,T k  y d Sim jk | s dj  s dk | 2 L sub = 1 2 (cid:88) d (cid:88) ( j,k ) | T j ,T k  y d Sub jk R ( s dj )(1  R ( s dk )) (3) where y d is the label set (annotated tags) of the document d . T is a list of all labels, where j and k are the indices of the list T , corresponding to the indices of nodes s dj and s dk in the output layer s d . R () is the rounding function for binary prediction, R ( s dj ) = 0 if S dj < 0 . 5 , otherwise R ( s dj ) = 1 . The similarity matrix Sim  (0 , 1) | T || T | indicates pairwise similarity between labels, the larger the value of Sim jk , the more similar the labels T j and T k are. Each element Sub jk in the subsumption matrix Sub  { 0 , 1 } | T || T | indicates whether the label T j is a child label of T k . Both the Sim and Sub matrix can be inferred from the training data or from external knowledge bases before training. In implementation, Sim (if thresh-olded) and Sub can be treated as sparse matrix to reduce computational complexity. We also used an adapted version of the loss regularisers in mini-batch training (the same set of label pairs that co-occurred within all documents in the same batch) to further to reduce computational complexity. The rationale is that the less the difference of the two outputs of the similar labels is, the lower the L sim . On the contrary, for output values not re-flecting the label similarity, i.e. large | s dj  s dk | 2 when Sim jk is close to 1, the error will be penalised with higher L sim . Given a document and a subsumption pair of labels, if the child label is used for annotation, its parent label has a relatively higher chance being used as well. In L sub , if a subsumption relation < T j  T k > presents in the label set y d , the case that the parent label T k is predicted as false, i.e. R ( s dk ) = 0 , when its child label T j is predicted as true, i.e. R ( s dj ) = 1 , will be penalised. Such a case will result in a positive penalty, while the penalty will be 0 in all other cases. Thus, L sim constrains similar labels to have similar outputs, while L sub reinforces each co-occurring subsumption pair to satisfy the dependency of the parent label on the child label. We evaluate our proposed approach for automated social annotation on two representative open datasets in social tagging, Bibsonomy 1 (academ-ic publication annotation) and Zhihu 2 (general do-main social question annotation). For Bibsonomy, we used the cleaned dataset from (Dong et al., 1 https://www.kde.cs.uni-kassel.de/bibsonomy/dumps 2 https://biendata.com/competition/zhihu/ Bibsonomy Precision Recall F 1 Score Time/Fold Bi-GRU .522  .020  .217  .016  .306  .019  1480  92s HAN .572  .008  .246  .012  .344  .013  1164  52s JMAN-s-tg .591  .010 .269  .006  .370  .007  1075  87s JMAN-s-att .586  .009 .269  .005  .369  .006  968  81s JMAN-s .586  .004 .282  .005 .380  .005 894  55s JMAN .592  .009 .284  .006 .384  .007 1044  73s  Paired t-tests at 95 percent significance level against the JMAN model. Table 1: Comparison Results on the Bibsonomy dataset Zhihu Precision Recall F 1 Score Time/Fold Bi-GRU .238  .011  .154  .009  .187  .010  1455  69s HAN .257  .012 .167  .010  .203  .011  1387  78s JMAN-s-tg .257  .005 .175  .003  .208  .006  1220  81s JMAN-s-att .254  .007  .174  .005  .207  .005  1275  99s JMAN-s .257  .008 .177  .005 .210  .007 1147  44s JMAN .260  .006 .179  .003 .212  .004 1135  52s  Paired t-tests at 95 percent significance level against the JMAN model.  Paired t-tests at 90 percent significance level against the JMAN model. Table 2: Comparison Results on the Zhihu dataset 2017) and further selected the tags related to Computer Sciences according to the ACM Computing Classification System 3 and selected the document that have both title and abstract (content); for Zhihu, we randomly sampled around 100,000 questions from the original data dump. The cleaned Bibsonomy dataset has 12,101 documents, 17,619 vocabularies and 5,196 labels; the average number of labels per document is 11.59. The sample Zhihu dataset has 108,168 documents (questions), 62,519 vocabularies and 1,999 labels; the average number of labels per document is 2.45. To calculate Sim , we used cosine similarity, normalised to between 0 and 1, of self-trained skip-gram embedding (Mikolov et al., 2013) on all label sets in each dataset. To obtain Sub , about subsumption relations, for Bibsonomy, we resorted to an external knowledge source Microsoft Concept Graph 4 for label mapping and semantic grounding; for Zhihu, we used the provided crowd-sourced label subsumption relations. We tuned the  1 and  2 in L based on 10-fold cross-validation 5 . We implemented the proposed Joint Multi-label Attention Network (JMAN) model in Figure 1 3 https://www.acm.org/publications/class-2012 4 https://concept.research.microsoft.com/Home 5  1 ,  2 were tuned to 1e-4, 1e-1 for Bibsonomy and 1e-3, 1e-1 for Zhihu, respectively. on Tensorflow (Abadi et al., 2016) along with the baselines 6 based on brightmart's implementation 7 of TextRNN and HAN under the MIT license. Two strong baselines were chosen Bi-GRU (Schuster and Paliwal, 1997; Cho et al., 2014) and HAN (Yang et al., 2016; Hassan et al., 2018). Several variations of JMAN were also considered: (i) JMAN-s , the proposed model without semantic-based loss regularisers; (ii) JMAN-s-tg , the proposed model without semantic-based regularisers and title guided sentence-level attention, c d = [ c t , c a ] ; (iii) JMAN-s-att , the proposed model without semantic-based regularisers and the original sentence-level attention, c d = [ c t , c ta ] . We optimised the joint loss L using the Adam optimiser (Kingma and Ba, 2014) and set the number of hidden units as 100, learning rate as 0.01 and dropout rate as 0.5 (Srivastava et al., 2014) for all models. The batch sizes for Bibsonomy and Zhihu were set as 128 and 1,024, respectively. The sequence lengths of the title (also the length of each sentence) and the content were padded to 30 and 300 for Bibsonomy and 25 and 100 for Zhihu. Non-static input embedding for the title and the sentences were initialised as 100-dimension self-trained skip-gram embedding (Mikolov et al., 6 Our code and datasets are available at https://github. com/acadTags/Automated-Social-Annotation . 7 https://github.com/brightmart/text_ classification 2013). We decayed the learning rate by half when the loss on validation set increased and set an ear-ly stopping point when learning rate is below 2e-5. All experiments were run on a GPU server, N-VIDIA GeForce GTX 1080 Ti. We report the mean and the standard deviation of the testing results on models trained with 10-fold cross-validation. The cleaned user-generated tags, i.e. labels, for each dataset were taken as the ground truth and the widely used example-based metrics, Precision, Recall and F 1 score (God-bole and Sarawagi, 2004; Tsoumakas et al., 2010; Zhang and Zhou, 2014), were adopted. The average training time per fold was also recorded. The results with respect to the two datasets are presented in the Table 1 and 2 respectively. Our proposed JMAN model significantly outperforms Bi-GRU and HAN. In terms of F 1 , with the Bibsonomy dataset, the proposed JMAN model provides a 7.8% absolute increase (by 25.5%) over Bi-GRU and 4.0% (by 11.6%) over HAN; on the Zhihu dataset, our model is 2.5% absolutely (by 13.4%) better than Bi-GRU and 0.9% (by 4.4%) than HAN. This is mostly attributed to the boost of recall through modeling the title metadata and the title-guided attention mechanism. The JMAN model also converges (understands) much faster than HAN with around 10.3% (for Bibsonomy) and 18.2% (for Zhihu) less training time per fold and converges even faster than Bi-GRU (by 29.5% and 22.0% for the Bibsonomy and Zhihu dataset in terms of training time, respectively). Recall and F 1 score drop significantly, with training time increased, when the title-guided or the original sentence-level attention is removed. Adding semantic-based loss regularisers further boosts the precision, recall and F 1 of the model. We also noticed that, compared to the results on the Bibsonomy dataset, the improvement on the Zhihu dataset with the proposed model is less sig-nificant. This may be related to the characterstics of the dataset: Zhihu has shorter texts (padded to 1/3 of the Bibsonomy dataset), more vocabularies (over 3 folds), less number of labels (about 40%) and less average number of labels per document (about 1/5) than the Bibsonomy dataset. This would warrant further study on the datasets and on validating the model with datasets from other social media platforms. We proposed a parallelled two-layer attention network for text annotation based on user-generated tags. It models the behaviour how human users read and understand document with the title-guided attention mechanism and leverages label semantics through two loss regularisers to constrain the network outputs. Experimental results show the effectiveness of this method with superi-or performance and training speed. This system can be applied to various types of social media platforms to support document organisation. Future studies will explore the possibility of applying the title-guided attention mechanism to other large datasets on major social media platforms. It is also interesting to see whether the semantic-based loss regularisers can be adapted to improve the performance of the recent pre-trained transferable deep learning models, such as the Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018). We thank all the anonymous reviewers for their constructive feedback. The implementation is based on brightmart's TextRNN and Hierarchical Attention Network under the MIT license 8 . This research is funded by the Research Development Fund at Xi'an Jiaotong-Liverpool University, contract number RDF-14-01-10 and partially supported by the following: The National Natural Science Foundation of China under no. 61876155; The Natural Science Foundation of the Jiangsu Higher Education Institutions of China under no. 17KJD520010; Suzhou Science and Technology Program under no. SYG201712, SZS201613; Natural Science Foundation of Jiangsu Province BK20181189; Key Program Special Fund in XJTLU under no. KSF-A-01, KSF-P-02, and KSF-E-26.
objective abstain abstain abstain method objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain objective abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method other method abstain abstain abstain method abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain other other other other other