Recent work in Dialogue Act classification has treated the task as a sequence labeling problem using hierarchical deep neural networks. We build on this prior work by leveraging the effectiveness of a context-aware self-attention mechanism coupled with a hierarchical recurrent neural network. We conduct extensive evaluations on standard Dialogue Act classification datasets and show significant improvement over state-of-the-art results on the Switchboard Dialogue Act (SwDA) Corpus. We also investigate the impact of different utterance-level representation learning methods and show that our method is effective at capturing utterance-level semantic text representations while maintaining high accuracy. Dialogue Acts (DAs) are the functions of utterances in dialogue-based interaction (Austin, 1975). A DA represents the meaning of an utterance at the level of illocutionary force, and hence, constitutes the basic unit of linguistic communication (Searle, 1969). DA classification is an important task in Natural Language Understanding, with applications in question answering, conversational agents, speech recognition, etc. Examples of DAs can be found in Table 1. Here we have a conversation of 7 utterances between two speakers. Each utterance has a corresponding label such as Question or Backchannel . Early work in this field made use of statistical machine learning methods and approached the task as either a structured prediction or text classification problem (Stolcke et al., 2000; Ang et al., 2005; Zimmermann, 2009; Surendran and Levow, 2006). Many recent studies have proposed deep learning models for the DA classification task with promising results (Lee and Dernoncourt, 2016; Khanpour et al., 2016; Ortega and Speaker Utterance DA label A Okay. Other A Um, what did you do this weekend? Question B Well, uh, pretty much spent most of my time in the yard. Statement B [Throat Clearing] Non Verbal A Uh-Huh. Backchannel A What do you have planned for your yard? Question B Well, we're in the process of, revitalizing it. Statement Table 1: A snippet of a conversation sample from the SwDA Corpus. Each utterance has a corresponding dialogue act label. Vu, 2017). However, most of these approaches treat the task as a text classification problem, treating each utterance in isolation, rendering them unable to leverage the conversation-level contextual dependence among utterances. Knowing the text and/or the DA labels of the previous utterances can assist in predicting the current DA state. For instance, in Table 1, the Answer or Statement dialog acts often follow Question type utterances. This work draws from recent advances in NLP such as self-attention, hierarchical deep learning models, and contextual dependencies to produce a dialogue act classification model that is effective across multiple domains. Specifically, we propose a hierarchical deep neural network to model different levels of utterance and dialogue act semantics, achieving state-of-the-art performance on the Switchboard Dialogue Act Corpus. We demonstrate how performance can improve by leveraging context at different levels of the model: previous labels for sequence prediction (using a CRF), conversation-level context with self-attention for utterance representation learning, and character embeddings at the word-level. Finally, we explore different ways to learn effective utterance representations, which serve as the building blocks of our hierarchical architecture for DA classification. A full review of all DA classification methods is outside the scope of the paper, thus we focus on two main classes of approaches which have dominated recent research: those that treat DA classification as a text classification problem, where each utterance is classified in isolation, and those that treat it as a sequence labeling problem. Text Classification : Lee and Dernoncourt (2016) build a vector representation for each utterance, using either a CNN or RNN, and use the preceding utterance(s) as context to classify it. Their model was extended by Khanpour et al. (2016) and Ortega and Vu (2017). Shen and Lee (2016) used a variant of the attention-based encoder for the task. Ji et al. (2016) use a hybrid architecture, combining an RNN language model with a latent variable model. Sequence Labeling : Kalchbrenner and Blunsom (2013) used a mixture of sentence-level CNNs and discourse-level RNNS to achieve state-of-the-art results on the task. Recent works (Li and Wu, 2016; Liu et al., 2017) have increasingly employed hierarchical architectures to learn and model multiple levels of utterance and DA dependencies. Kumar et al. (2018), Chen et al. (2018) and Tran et al. (2017) used RNN-based hierarchical neural networks, using different combinations of techniques like last-pooling or attention mechanism to encode sentences, coupled with CRF decoders. Chen et al. (2018) achieved the highest performance to date on the two datasets for this task. Our work extends these hierarchical models and leverages a combination of techniques proposed across these prior works (CRF decoding, contextual attention, and character-level word embeddings) with self-attentive representation learning, and is able to achieve state-of-the-art performance. The task of DA classification takes a conversation C as input, which is a varying length sequence of utterances U = { u 1 , u 2 , ...u L } . Each utterance u i  U , in turn, is a sequence of varying lengths of words { w 1 i , w 2 i , ..., w N i i } , and has a corresponding target label y i  Y . Hence, each conversation (i.e. a sequence of utterances) is mapped to a corresponding sequence of target (cid:6932) 21 (cid:6969)  21 (cid:6969)  21 (cid:6969)  22 (cid:6969)  22 (cid:6969)  23 (cid:6969)  23 (cid:6966) 21 (cid:6969) 21 (cid:6969) 22 (cid:6969) 23 (cid:6968)  1 (cid:6968)  1 (cid:6968)  2 (cid:6968)  2 (cid:6968)  3 (cid:6968)  3 (cid:6968) 1 (cid:6968) 2 (cid:6968) 3 (cid:6986) 1 (cid:6922) 1 (cid:6922) 2 (cid:6922) 3 ... ... (cid:6922) n (cid:6897) (cid:6966) 22 (cid:6966) 23 how Words Embeddings Bi-Directional RNN Concatenate (cid:6986) 2 (cid:6986) 3 Context-aware Self-Attention Fully Connected Bi-Directional RNN Concatenate CRF (cid:6969) 2 (cid:6969) 1 (cid:6969) 3 (cid:6964) 21 (cid:6932) 22 are (cid:6964) 22 (cid:6932) 23 you (cid:6964) 23 Figure 1: Model Architecture labels Y = { y 1 , y 2 , ..., y L } , which represents the DAs associated with the corresponding utterances. Figure 1 shows the overall architecture of our model, which involves three main components: (1) an utterance-level RNN that encodes the information within the utterances at the word and character-level; (2) a context-aware self-attention mechanism that aggregates word representations into utterance representations; and (3) a conversation-level RNN that operates on the utterance encoding output of the attention mechanism, followed by a CRF layer to predict utterance labels. We describe them in detail below. For each word in an utterance, we combine two different word embeddings: GloVe (Pennington et al., 2014) and pre-trained ELMo representations (Peters et al., 2018) with fine-tuned task-specific parameters, which have shown superior performance in a wide range of tasks. The word embedding is then concatenated with its CNN-based 50-D character-level embedding (Chiu and Nichols, 2016; Ma and Hovy, 2016) to get the complete word-level representations. The motivation behind incorporating subword-level information is to infer the lexical features of utterances and named entities better. The word representation layer is followed by a bidirectional GRU (Bi-GRU) layer. Concatenating the forward and backward outputs of the Bi-GRU generates the utterance embedding that serves as input to the utterance-level context-aware self-attention mechanism which learns the final utterance representation. Self-attentive representations encode a variable-length sequence into a fixed size, using an attention mechanism that considers different positions within the sequence. Inspired by Tran et al. (2017), we use the previous hidden state from the conversation-level RNN (Section 3.3), which provides the context of the conversation so far, and combine it with the hidden states of all the constituent words in an utterance, into a self-attentive encoder (Lin et al., 2017), which computes a 2 D representation of each input utterance. We follow the notation originally presented in Lin et al. (2017) to explain our modification of their self-attentive sentence representation below. An utterance u i , which is a sequence of n words { w 1 i , w 2 i , ...w ni } , is mapped into an embedding layer, resulting in a d -dimensional word embedding for every word. It is then fed into a bidirectional-GRU layer, whose hidden state outputs are concatenated at every time step. H i represents the n GRU outputs of size 2 u ( u is the number of hidden units in a unidirectional GRU). Here, W s 1 is a weight matrix with a shape of d a  2 u , W s 2 is a matrix of parameters of shape r  d a , where r and d a are hyperparameters we can set arbitrarily, and W s 3 is a parameter matrix of shape d a  k for the conversational context, where k is another hyperparameter that is the size of a hidden state in the conversation-level RNN (size of  g i  1 ), and b is a vector representing bias. Equation 5 can then be treated as a 2-layer MLP with bias, with d a hidden units, W s 1 , W s 2 and W s 3 as weight parameters. The scores S i are mapped into a probability matrix A i by means of a softmax function: A i = softmax ( S i ) (6) which is then used to obtain a 2-d representation M i of the input utterance, using the GRU hidden states H i according to the attention weights provided by A i as follows: M i = A i H i (7) This 2-d representation is then projected to a 1-d embedding (denoted as h i ), using a fully-connected layer. The conversation-level GRU then operates over this 1-d utterance embedding, and hence, we can represent g i as:  g i =  GRU ( h i ,  g i  1 ) (8)  g i =  GRU ( h i ,  g i +1 ) (9) g i = concat (  g i ,  g i ) (10) g i then provides the conversation-level context used to learn the attention scores and 2-d representation ( M i +1 ) for the next utterance in the conversation ( h i + 1 ). The utterance representation h i from the previous step is passed on to the conversation-level RNN, which is another bidirectional GRU layer used to encode utterances across a conversation. The hidden states  g i and  g i (Figure 1) are then concatenated to get the final representation g i of each utterance, which is further propagated to a linear chain CRF layer. The CRF layer considers the correlations between labels in context and jointly decodes the optimal sequence of utterance labels for a given conversation, instead of decoding each label independently. We evaluate the classification accuracy of our model on the two standard datasets used for DA classification: the Switchboard Dialogue Act Corpus (SwDA) (Jurafsky et al., 1997) consisting of 43 classes, and the 5-class version of the ICSI Meeting Recorder Dialogue Act Corpus (MRDA) introduced in (Ang et al., 2005). For both datasets, Dataset | C | | V | Train Validation Test MRDA 5 12k 78k 16k 15k SwDA 43 20k 193k 23k 5k Table 2: Number of utterances by dataset. we use the train, validation and test splits as de-fined in Lee and Dernoncourt (2016). Table 2 shows the statistics for both datasets. They are highly imbalanced in terms of class distribution, with the DA classes Statement-non-opinion and Acknowledge/Backchannel in SwDA and Statement in MRDA making up over 50% of the labels in each set. We compare the classification accuracy of our model against several other recent methods (Ta-ble 3). 1 Four approaches (Chen et al., 2018; Tran et al., 2017; Ortega and Vu, 2017; Shen and Lee, 2016) use attention in some form to model the conversations, but none of them have explored self-attention for the task. The last three use CRFs in the final layer of sequence labeling. Only one other method (Chen et al., 2018) uses character-level word embeddings. All models and their variants were trained ten times and we report the average test performance. Our model outperforms state-of-the-art methods by 1.6% on SwDA, the primary dataset for this task, and comes within 0.6% on MRDA. It also beats a TF-IDF GloVe baseline (described in Section 5.2) by 16.4% and 12.2%, respectively. The improvements that the model is able to make over the other methods are significant, however, the gains on MRDA still fall short of the state-of-the-art by 0.6%. This can mostly be attributed to the conversation/context lengths and label noise at the conversation level. Conversations in MRDA (1493 utterances on average) are significantly longer than in SwDA (271 utterances on av-erage). In spite of having nearly 12% the number 1 Contemporaneous to this submission, (Li et al., 2018; Wan et al., 2018; Ravi and Kozareva, 2018) proposed different approaches for the task. We do not focus on them here per NAACL 2019 guidelines, however note that our system outperforms the first two. (Ravi and Kozareva, 2018) bypasses the need for complex networks with huge parameters but its overall accuracy is 4.2% behind our system, despite being 0.2% higher on SwDA. of labels (5 vs 43) compared to SwDA, MRDA has 6 times the normalized label entropy in its data. Consequently, due to the noise in label dependencies, and hence, in the inherent conversational structure, the model is not able to yield as big of a gain on the MRDA as it does on the SwDA. Consequently, learning long-range dependencies is a challenge because of noisier and longer path lengths in the network. This is illustrated in Figures 2 and 3, which show for every class, the variation between the entropy of the previous label in a conversation, and the accuracy of that class. MRDA was found to have a high negative correlation 2 (-0.68) between previous label entropy and accuracy, indicating the impact of label noise, which was compounded by longer conversations. On the other hand, SwDA was found to have a low positive correlation (+0.22), which could be compensated by significantly shorter conversations. One of the primary motivations for this work was to investigate whether one can improve performance by learning better representations for utterances. To address this, we retrained our model by replacing the utterance representation learning (utterance-level RNN + context-aware self-attention) component with various sentence representation learning methods (either pre-training them or learning jointly), and feeding them into the conversation-level recurrent layers in the hierarchical model, so that the performance is indicative of the quality of utterance representations. There are three main categories of utterance representation learning approaches: (i) the baseline which uses a TF-IDF weighted sum of GloVe word embeddings; (ii) pre-trained on cor-2 Pearson's r Method SwDA MRDA Baseline TF-IDF GloVe 66.5 78.7 Pre-trained on Corpus Skip Thought Vectors 72.6 82.8 Paragraph vectors 72.5 82.6 Joint Learning RNN-Encoder 74.8 85.7 Bi-RNN-LastState 76.2 85.4 Bi-RNN-MaxPool 77.6 86.7 CNN 76.9 84.5 Bi-RNN + Attention 80.1 87.7 + Context 81.8 89.2 Bi-RNN + Self-attention 81.1 88.6 + Context 82.9 91.1 Table 4: Performance of utterance representation methods when integrated with the hierarchical model pus, where we first learn utterance representations on the corpus using Skip-Thought Vectors (Kiros et al., 2015) and Paragraph Vectors (Le and Mikolov, 2014), and then use them with the rest of the model; (iii) jointly learned with the DA classification task. Table 4 describes the performance of different utterance representation learning methods when combined with the overall architecture on both datasets. Introducing the word-level attention mechanism (Yang et al., 2016) enables the model to learn better representations by attending to more informative words in an utterance, resulting in better performance (Bi-RNN + Attention). The self-attention mechanism (Bi-RNN + Self-attention) leads to even greater overall improvements. Adding context information (previous recurrent state of the conversation) boosts performance significantly. A notable aspect of our model is how contextual information is leveraged at different levels of the sequence modeling task. The combination of conversation-level contextual states for utterance-representation learning (+ Context) and a CRF at the conversation level to further inform conversation sequence modeling, leads to a collective performance improvement. This is particularly pronounced on the SwDA dataset: the two variants of the context-aware attention models (Bi-RNN + Attention + Context and Bi-RNN + Self-attention + Context) have significant performance gains. We developed a model for DA classification with context-aware self-attention, which significantly outperforms earlier models on the commonly-used SwDA dataset and is very close to state-of-the-art on MRDA. We experimented with different utterance representation learning methods and showed that utterance representations learned at the lower levels can impact the classification performance at the higher level. Employing self-attention, which has not previously been applied to this task, enables the model to learn richer, more effective utterance representations for the task. As future work, we would like to experiment with other attention mechanisms such as multihead attention (Vaswani et al., 2017), directional self-attention (Shen et al., 2018a), block self-attention (Shen et al., 2018b), or hierarchical attention (Yang et al., 2016), since they have been shown to address the limitations of vanilla attention and self-attention by either incorporating information from different representation subspaces at different positions to capture both local and long-range context dependencies, encoding temporal order information, or by attending to context dependencies at different levels of granularity. The authors would like to thank Dimitris Alikanio-tis, Maria Nadejde and Courtney Napoles for their insightful discussions, and the anonymous reviewers for their helpful comments.
abstain method result objective abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method objective objective objective method other other other other other other other other objective abstain abstain other other abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain other abstain other method abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result abstain result other