A CONTEXT-AWARE ATTENTION NETWORK FOR INTERACTIVE QUESTION ANSWERING
We develop a new model for Interactive Question Answering (IQA), using GatedRecurrent-Unit recurrent networks (GRUs) as encoders for statements and questions, and another GRU as a decoder for outputs. Distinct from previous work, our approach employs context-dependent word-level attention for more accurate statement representations and question-guided sentence-level attention for better context modeling. Employing these mechanisms, our model accurately understands when it can output an answer or when it requires generating a supplementary question for additional input. When available, user’s feedback is encoded and directly applied to update sentence-level attention to infer the answer. Extensive experiments on QA and IQA datasets demonstrate quantitatively the effectiveness of our model with significant improvement over conventional QA models.
1 INTRODUCTION
The ultimate goal of Question Answering (QA) research is to build intelligent systems capable of naturally communicating with humans, which poses a major challenge for natural language processing and machine learning. Inspired by recent success of sequence-to-sequence models with an encoder-decoder framework (Sutskever et al., 2014; Cho et al., 2014), researchers have attempted to apply variants of such models with explicit memory and attention to QA tasks, aiming to move a step further from machine learning to machine reasoning (Sainbayar et al., 2015; Kumar et al., 2016; Xiong et al., 2016). Similarly, all these models employ encoders to map statements and questions to fixed-length feature vectors, and a decoder to generate outputs. Empowered by the adoption of memory and attention, they have achieved remarkable success on several challenging datasets, including the recently acclaimed Facebook bAbI dataset.
However, previous models suffer from the following important limitations. First, they fail to model context-dependent meaning of words. Different words may have different meanings in different contexts, which increases the difficulty of extracting the essential semantic logic flow of each sentence in different paragraphs. Second, many existing models only work in ideal QA settings and fail to address the uncertain situations under which models require additional user input to gather complete information to answer a given question. As shown in Table 1, the example on the left is an ideal QA problem. We can clearly understand what the question is and then locate the relevant sentences to generate the answer. However, it is hard to answer the question in the right example, because there are two types of bedrooms mentioned in the story and we do not know which bedroom the user refers to. These scenarios with incomplete information naturally appear in human conversations, and thus, effectively handling them is a key capability of intelligent QA models.
To address the challenges presented above, we propose a Context-aware Attention Network (CAN) to learn fine-grained representations for input sentences, and develop a mechanism to interact with the user for comprehensively understanding a given question. Specifically, we employ two-level attention applied at word level and sentence level to compute representations of all input sentences.
∗Most of this work was done when the author was an intern at NEC Labs America.
The context information extracted from the input story is allowed to influence the attention over each word, and governs the word semantic meaning contributing to a sentence representation. In addition, an interactive mechanism is activated to generate a supplementary question for the user when the model feels that it does not have complete information to answer a given question. User’s feedback is then encoded and exploited to attend over all input sentences to infer the answer. Our proposed model CAN can be viewed as an encoder-decoder approach augmented with two-level attention and an interactive mechanism, rendering our model self-adaptive, as illustrated in Figure 1.
Our contributions in this paper are as follows: (i) We develop a new encoder-decoder model called CAN for question answering with two-level attention. Due to the new attention mechanism, our model avoids the necessity of multiple-hop attention, required by previous QA models, and knows when it can readily output an answer and when it needs additional information. (ii) We augment the encoder-decoder framework for QA with an interactive mechanism for handling user’s feedback, which immediately changes sentence-level attention to infer the final answer without additional model training. (iii) We introduce a new dataset based on the bAbI dataset, namely ibAbI, for IQA tasks. (iv) Extensive experiments show that our approach outperforms state-of-the-art models on both QA and IQA datasets. Specifically, our approach achieves 40% improvement over traditional QA models (e.g., MemN2N and DMN+) on IQA datasets.
2 RELATED WORK
Recent work on QA has been heavily influenced by research on various models with attention and/or memory. Most of these models employ an encoder-decoder framework, and have been successfully applied to image classification (Seo et al., 2016), image captioning (Xu et al., 2015; Mnih et al., 2014), machine translation (Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015), document classification (Yang et al., 2016), and textual/visual QA (Sainbayar et al., 2015; Yang et al., 2015; Lu et al., 2016; Kumar et al., 2016; Xiong et al., 2016). For textual QA in the form of statementsquestion-answer triplets, Sainbayar et al. (2015) utilizes an external memory module. It maps each input sentence to an input representation space regarded as a memory component. The output representation is calculated by summarizing over input representations with different attention weights. This single-layer memory can be extended to multi-layer memory by reasoning the content and the question multiple times. Instead of simply stacking the memory layers, Kumar et al. (2016) have introduced a dynamic memory network (DMN) to update the memory vectors through a modified GRU, in which the gate weight is trained in a supervised fashion. To improve DMN by train-
ing without supervision, Xiong et al. (2016) encode input sentences with a bidirectional GRU and then utilize an attention-based GRU to summarize these input sentences. Neural Turing Machine (NTM) (Graves et al., 2014), a model with content and location-based memory addressing mechanisms, has also been used for QA tasks recently. There is other recent work about QA using external resources (Wu et al., 2015; Fader et al., 2014; Savenkov & Emory, 2016; Hermann et al., 2015; Golub & He, 2016), and exploring dialog tasks (Weston, 2016; Bordes & Weston, 2016).
Our model in this paper also addresses textual QA in the form of statements-question-answer triplets, but it differs from prior work in two aspects. First, in our attention network, the word attention are context-dependent for generating accurate sentence representations and the sentence attention are question-guided for generating context representation. Second, this new attention mechanism helps our model understand when it can readily output an answer and when it can generate a supplementary question for activating the user interaction. Incorporating user’s feedback does not require additional model training and this property makes our model highly self-adaptive.
3 GATED RECURRENT UNIT NETWORKS
Gated Recurrent Unit (GRU) (Cho et al., 2014) is the basic building block of our model for IQA. GRU has been widely adopted for many NLP tasks, such as machine translation (Bahdanau et al., 2015) and language modeling (Zaremba et al., 2014). GRU improves Long Short-term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) by removing the cell component and making each hidden state adaptively capture the dependencies over different time scales using reset and update gates. For each time step t with input xt and previous hidden state ht−1, we compute the updated hidden state ht = GRU(ht−1,xt) by,
rt = σ(Urx t +Wrh t−1 + br), z t = σ(Uzx t +Wzh t−1 + bz),
h̃t = tanh(Uhx t +Wh(r t ht−1) + bh), ht = zt ht−1 + (1− zt) h̃t,
where σ is the sigmoid activation function, is an element-wise product, Ur,Uz,Uh ∈ RK×D, Wr,Wz,Wh ∈ RK×K , br,bz,bh ∈ RK×1, K is the hidden size and D is the input size.
4 CONTEXT-AWARE ATTENTION NETWORK
In this section, we first illustrate the proposed model CAN (§ 4.1), including the question module (§ 4.2), the input module(§ 4.3) and the answer module (§ 4.4). We then describe each of these modules in detail. Finally, we elaborate the training procedure of CAN (§ 4.5).
4.1 FRAMEWORK
Given a story represented by N input sentences (or statements), i.e., (l1, · · · , lN ), and a question q, our goal is to generate an answer a. Each sentence lt includes a sequence of Nt words, denoted as (wt1, · · · , wtNt), and a question with Nq words is represented as (w q 1, · · · , w q Nq
). Let V denote the size of dictionary, including the words from each lt, q and a, and end-of-sentence (EOS) symbols.
The whole framework of our model is shown in Figure 2, consisting of the following three key parts:
• Question Module: The question module encodes the target question into a vector representation.
• Input Module: The input module encodes a set of input sentences into a vector representation.
• Answer Module: The answer module generates an answer based on the outputs of question and input modules. Unlike traditional QA models, it has two choices, either to output an answer immediately or to interact with the user for further information. Hence, if the model lacks sufficient evidence for answer prediction based on the existing knowledge at current timestamp, an interactive mechanism is enabled. Specifically, the model generates a supplementary question, and the user needs to provide a feedback, which is utilized to estimate an answer.
4.2 QUESTION MODULE
Suppose a question is a sequence of Nq words, we encode each word wj into a Kw-dimensional vector space xqj using an embedding matrix Ww ∈ RKw×V , i.e., x q j = Ww[wj ], where [wj ] is a one-hot vector associated with word wj . The sequence order within a sentence significantly affects each word’s semantic meaning due to its dependence on the previous words. Thus, a GRU is employed by taking each word vector xqj as input and updating the hidden state g q j ∈ RKh×1 as:
gqj = GRUw(g q j−1,x q j), (1)
where the subscript of GRU is used to distinguish other GRUs used in the following sections. The hidden state gqj can be regarded as the annotation vector of word wj by incorporating the word order information. We also explore a variety of encoding schema, such as LSTM and RNN. However, LSTM is prone to overfitting due to large number of parameters, and RNN has a poor performance because of exploding and vanishing gradients (Bengio et al., 1994).
In addition, each word contributes differently to the representation of a question. For example, in a question ‘Where is the football?’, ‘where’ and ‘football’ play a critical role in summarizing this sentence. Therefore, an attention mechanism is introduced to generate a question representation by focusing on the important words for their semantic meaning. A positive weight γj is placed on each word to indicate the relative importance of contribution to the representation of the question. Specifically, this weight is measured as the similarity of corresponding word annotation vector gj and a word level latent vector v ∈ RKh×1 for question which is jointly learned during the training process. The question representation u ∈ RKc×1 is then generated by a weighted summation of the word annotation vectors and corresponding important weights, where we also use one-layer MLP to transfer it from sentence-level space into context-level space,
γj = softmax(v Tgqj ), (2) u = Wch Nq∑ j=1 γjg q j + b (q) c , (3)
where softmax is defined as softmax(xi) = exp(xi)∑ j′ exp(xj′ ) , Wch ∈ RKc×Kh , and b(q)c ∈ RKc×1.
4.3 INPUT MODULE
Input module aims at generating a representation for input sentences, including a sentence encoder and a context encoder. Sentence encoder computes a sentence representation, and context encoder calculates a representation of input sentences, both of which are introduced in the following sections.
4.3.1 SENTENCE ENCODER
For each input sentence lt, containing a sequence ofNt words (w1, · · · , wNt), similar to the question module, each word wi is embedded into word space xti ∈ RKw×1 with the embedding matrix Ww, and a recurrent neural network is used to capture the context information from the words which have already been generated in the same sentence. Let hti ∈ RKh×1 denote the hidden state which can be interpreted as the word annotation in the input space. A GRU retrieves each word annotation by taking word vector as input and relying on previous hidden state,
hti = GRUw(h t i−1,x t i). (4)
In Eq.(4), each word annotation vector takes the sequence order into consideration to learn its semantic meaning based on previous information within a sentence through a recurrent neural network. A question answering system is usually given multiple input sentences which often form a story together. A single word has different meaning in the different stories. Learning a single sentence context at which a word is located is insufficient to understand the meaning of this word, in particular when the sentence is placed in a story context. In other words, only modeling a sequence of words prior to a word within a sentence may lose some important information which results in the failure of the generation of sentence representation. Hence, we take the whole context into account as well to appropriately characterize each word and well understand this sentence’s meaning. Suppose st−1 ∈ RKc×1 is the annotation vector of previous sentence lt−1, which will be introduced in the next section. To incorporate context information generated by previous sentences, we feed word annotation hti and previous sentence annotation st−1 through a two-layer MLP, where a context-aware word vector eti ∈ RKc×1 is obtained as follows:
eti = σ(Weetanh(Wesst−1 +Wehh t i + b (1) e ) + b (2) e ), (5)
where Wee,Wes ∈ RKc×Kc and Weh ∈ RKc×Kh are weight matrices, and b(1)e ,b(2)e ∈ RKc×1 are the bias terms. It is worth noting that st−1 is dependent on its previous sentence. Recursively, this sentence relies on its previous one as well. Hence, our model is able to encode the previous context. In addition, the sentence representation will focus on those words which are able to address the question. Inspired by this intuition, another word level attention mechanism is introduced to attend informative words about the question for generating a sentence’s representation. As the question representation is utilized to guide the word attention, a positive weight αti associated with each word is computed as the similarity of the question vector u and the corresponding context-aware word vector eti. Then the sentence representation yt ∈ RKh×1 is generated by aggregating the word annotation vectors with different weights,
αti = softmax(u Teti), yt = Nt∑ i=1 αtih t i. (6)
4.3.2 CONTEXT ENCODER
Suppose a story is comprised of a sequence of sentences, i.e., (l1, · · · , lN ), each of which is encoded as a Kh-dimensional vector yt through a sentence encoder. As input sentences have a sequence order, simply using their sentence vectors for context generation cannot effectively capture the entire context of the sequence of sentences. To address this issue, a sentence annotation vector is introduced to capture the previous context and this sentence’s own meaning using a GRU. Given the sentence vector yt and the state st−1 of previous sentence, we get annotation vector st ∈ RKc×1 as:
st = GRUs(st−1,yt). (7)
A GRU can learn a sentence’s meaning based on previous context information. However, just relying on GRU at sentence level using simple word embedding vectors makes it difficult to learn the precise semantic meaning for each word in the story. Hence, we introduce a context-aware attention mechanism shown in Eq.(5) to properly encode each word for the generation of sentence representation, which guarantees that each word is reasoned under the specific context.
Once the sentence annotation vectors (s1, · · · , sN ) are obtained as described above, a sentence level attention mechanism is enabled to emphasize those sentences that are highly relevant to the question. We can estimate the attention weight βt with the similarity of the question and the
corresponding sentence. Hence, the context representation m is retrieved by summing over all sentence representations associated with corresponding attention weights, and given by:
βt = softmax(u T st), (8) m = N∑ t=1 βtst. (9)
Similar to bidirectional RNN, our model can be extended to use another sentence-level GRU that moves backward through time beginning from the end of the sequence.
4.4 ANSWER MODULE
The answer module utilizes a decoder to generate an answer, where it has two output cases according to the understanding ability of both the question and the context. One is to generate the answer immediately after receiving the context and question information. Another one is to generate a supplementary question and then use the user’s feedback to predict the answer. This process is taken by an interactive mechanism.
4.4.1 ANSWER GENERATION
Given the question representation u and the context representation m, another GRU is used as the decoder to generate a sentence as the answer. To fuse u and m together, we sum these vectors rather than concatenating them to reduce the total number of parameters. Suppose x̂k−1 ∈ RKw×1 is the predicted word vector in last step, GRU updates the hidden state zk ∈ RKo×1 as follows,
x̂k Ww= softmax(Wodzk + bo), zk = GRUd(zk−1, [m+ u; x̂k−1]) (10)
where Ww= denotes the predicted word vector through the embedding matrix Ww. Note that we require that each sentence ends with a special EOS symbol, including question mask and period symbol, which enables the model to define a distribution over sentences of all possible lengths.
Output Choices. In practice, the system is not aways able to answer question immediately based on its current knowledge due to the lack of some crucial information bridging the gap between question and context knowledge, i.e., incomplete issue. Therefore, we allow the decoder to make a binary choice, either to generate an answer immediately, or to enable an interactive mechanism. Specifically, if the model has sufficiently strong evidence for a successful answer prediction based on the well-learned context representation and question representation, the decoder will directly output the answer. Otherwise, the system generates a supplementary question for user, where an example is shown in Table 2. At this time, this user needs to offer a feedback which is then encoded to update the sentence-level attentions for answer generation. This procedure is our interactive mechanism.
The sentence generated by the decoder ends with a special symbol, either a question mask or a period symbol. Hence, this special symbol is utilized to make a decision. In other words, if EOS symbol is a question mask, the generated sentence is regarded as a supplementary question and an interactive mechanism is enabled; otherwise the generated sentence is the estimated answer and the prediction task is done. In the next section, we will introduce the details of interactive mechanism.
4.4.2 INTERACTIVE MECHANISM
The interactive process is summarized as follows: 1) The decoder generates a supplementary question; 2) The user provides a feedback; 3) The feedback is used for answer prediction for the target question. Suppose the feedback contains a sequence of words, denoted as (wf1 , · · · , w f Nf ). Similar to the input module, each word wfd is embedded to a vector x f d through an embedding matrix
Ww. Then the corresponding annotation vector g f d ∈ RKh×1 is retrieved via a GRU by taking the embedding vector as input, and shown as follows:
gfd = GRUw(g f d−1,x f d). (11)
Based on the annotation vectors, a representation f ∈ RKh×1 can be obtained by a simple attention mechanism where each word is considered to contribute equally, and given by:
f = 1
Nf Nf∑ d=1 gfd . (12)
Our goal is to utilize the feedback representation f to generate an answer for the target question. The provided feedback improves the ability to answer the question by distinguishing the relevance of each input sentence to the question. In other words, the similarity of specific input sentences in the provided feedback make these sentences more likely to address the question. Hence, we refine the attention weight of each sentence shown in Eq.(9) after receiving the user’s feedback, given by,
r = tanh(Wrf f + b (f) r ), (13) βt = softmax(u T st + r T st) (14)
where Wrf ∈ RKc×Kh and b(f)r ∈ RKc×1 are the weight matrix and bias vector, respectively. Eq.(13) is an one-layer neural network to transfer feedback representation to context space. After obtaining the newly learned attention weights, we update the context representation using the softattention operation shown in Eq.(9). This updated context representation and question representation will be used as the input for decoder to generate an answer. Note that for simplifying the problem, we allow the decoder to only generate at most one supplementary question. In addition, one advantage of using the user’s feedback to update the attention weights of input sentences is that we do not need to re-train the encoder again once a feedback is entering the system.
4.5 TRAINING PROCEDURE
During training, all modules share an embedding matrix. There are three different GRUs employed for sentence encoding, context encoding and answer/supplementary question decoding. In other words, the same GRU is used to encode the question, input sentences and the user’s feedback. The second one is applied to generate context representation and the third one is used as decoder. Training can be treated as a supervised classification problem to minimize the cross-entropy error of the answer sequence and the supplementary question sequence.
5 EXPERIMENTS
In this section, we evaluate our approach with many baseline methods based on various datasets.
5.1 EXPERIMENTAL SETUP
Datasets. In this paper, we use two types of datasets to evaluate the performance of our approach. One is traditional QA dataset, where we use Facebook bAbI English 10k dataset (Weston et al., 2015). It contains 20 different types of tasks with emphasis on different forms of reasoning and induction. The second is the newly designed IQA dataset, where we extend bAbI to add interactive QA and denote it as ibAbI. Overall, we generate three ibAbI datasets based on task 1 (single supporting fact), task 4 (two argument relations), and task 7 (counting). Specifically, the former two datasets focus on solving ambiguous actors/objects problem, and the latter one is to ask further information that assists answer prediction. Table 3 shows three examples for our three ibAbI tasks.
In addition, we also mix IQA data and corresponding QA data together with different IQA ratios, where the IQA ratio is ranging from 0.3 to 1 (with step as 0.1) and denoted asRIQA. For example, in task 1, we randomly pick RIQA× 100 percent data from ibAbI task 1, and then randomly select the remaining data from bAbI task 1. RIQA = 1 indicates that the whole dataset only consists of IQA problems; otherwise (i.e., ranging from 0.3 to 0.9) it consists of both types of QA problems. Overall, we have three tasks for ibAbI dataset, and eight sub-datasets for each task. In the experiments, 10k examples are used as training and another 1k examples are used as testing.
Experiment Settings. We train our models using the Adam optimizer (Kingma & Ba, 2014). Xavier initialization is used for all parameters except for word embeddings, which utilize random uniform initialization ranging from − √ 3 to √ 3. The learning rate is set as 0.001. The grid search method is utilized to find optimal parameters, such as batch size and hidden size.
5.2 BASELINE METHODS
To demonstrate the effectiveness of our approach CAN, we compare it with the following models:
• DMN+: Xiong et al. (2016) improve Dynamic Memory Networks (Kumar et al., 2016) by using stronger input and memory modules, where a bidirectional GRU is adopted to generate representations for statements and a neural network is used to update episodic memory multiple times.
• MemN2N: This is an extension of Memory Network with weak supervision as proposed in Sainbayar et al. (2015). Here, an external memory module is used to encode the input statements and a recurrent attention mechanism is used to read the memory for answer prediction.
• EncDec: We extend the encoder-decoder framework (Cho et al., 2014) to solve QA tasks as a baseline method. Specifically, EncDec uses a GRU to encode statements and questions, the end of hidden states is used as context representation, and another GRU to generate the output.
5.3 PERFORMANCE OF QUESTION ANSWERING
In this section, we evaluate model’s ability for answer prediction based on traditional QA dataset (i.e., bAbI-10k). For this task, our model (denoted as CAN+QA) does not use the interactive mechanism. As the output answers for this dataset only contain a single word, we adopt test error rate as evaluation metric. For DMN+ and MemN2N methods, we select the best performance over bAbI dataset reported in (Xiong et al., 2016). The results of various models across 20 tasks are reported in Table 4. We summarize the main observations as follows:
• Our approach is better than all baseline methods in each individual task. For example, it reduces the error by 4% compared to DMN+ in task 17, and compared to MemN2N, it reduces 18.4% and 4.8% error in task 17 and 18 respectively. We can achieve a better result primarily because our approach can model the semantic logic flow for statements. Table 5 shows two examples in task 17 and 18, where MemN2N predicts incorrectly while CAN+QA can make correct predictions. In these two examples, the semantic logic determines the relationship between two objects mentioned in the question, such as chest and suitcase. In addition, Kumar et al. (2016) has shown that memory networks with multiple hops are better than the one with single hop. Our strong results illustrate that our approach has more accurate context modeling even without multiple hops.
• EncDec performs the worst amongst all models over all tasks. EncDec concatenates the statements and questions as a single input, resulting in the difficulty of training the GRU. For example, EncDec is not good on task 2 and 3 because these two tasks have longer inputs than other tasks.
• The results of DMN+ and MemN2N are much better than EncDec. It is not surprising that they can outperform EncDec, because they are specifically designed for question answering and do not suffer from the problem mentioned above by treating input sentences separately.
• All models perform poorly on task 16. Xiong et al. (2016) points out that MemN2N with a simple update for memory could achieve a near perfect error rate of 0.4 while a more complex method will lead to a much worse result. This shows that a sophisticated modeling method makes it
difficult to achieve a good performance in certain simple tasks with such limited data. This can be a possible reason for the poor performance of our model on this specific task as well.
In addition, different from MemN2N, we use a GRU to capture the semantic logic flow of input sentences, where the sentence-level attention can weaken the influence of unrelated sentences in a long story. Table 6 shows two examples of our results with long stories. From the attention weights, we can see our model can correctly search relevant sentences in a long story to address a question.
5.4 PERFORMANCE OF INTERACTIVE QUESTION ANSWERING
In this section, we evaluate the performance of various models based on IQA dataset (as described in Section 5.1). For testing, we simulate the interactive procedure by taking the predefined feedback as user’s input for the generated supplementary question, and then generating an answer. All baseline methods do not have interactive part, so they take both statements and question as input and then estimate an answer. We compare our approach (CAN+IQA) with baseline methods in terms of test error rate shown in Table 7. From the results, we can achieve the following conclusions:
• Our method significantly outperforms all baseline methods. Specifically, we can achieve 0% test error rate in task 1 and task 4 withRIQA = 1.0 ; while the best result of baseline methods can only get 40.5% test error rate. CAN+IQA benefits from more accurate context modeling, which allows it to correctly understand when to output an answer or require additional information. For those QA problems with incomplete information, it is necessary to gather the additional information from users. Randomly guessing may harm model’s performance, which makes conventional QA models difficult to converge. But our approach uses an interactive procedure to obtain user’s feedback and allows the model to provide the correct answer.
• For the baseline methods, DMN+ and MemN2N perform similarly and do better than EncDec. Their similar performance (which are worse than our approach) is due to the limitation that they could not learn the accurate meaning of statements and questions with limited resource and then have trouble training the models. But they are superior over EncDec as they treat each input sentence separately instead of modeling very long inputs.
In addition, we also quantitatively evaluate the quality of supplementary question generated by our approach where the details can be found in Appendix A.
5.5 QUALITATIVE ANALYSIS OF INTERACTIVE MECHANISM
In this section, we qualitatively show the attention weights over input sentences generated by our model on both QA and IQA data. We train our model (CAN+IQA) on task 1 of ibAbI dataset with QIQA = 0.9, and randomly select one IQA example from the testing data. Then we do the prediction on this IQA problem. In addition, we change this instance to a QA problem by replacing the question “Where is she?” with “Where is Sandra?”, and then do the prediction as well. The prediction results on both QA and IQA problems are shown in Table 8. From the results, we observe the following: 1) The attention that uses user’s feedback focuses on the key relevant sentence while the attention without feedback only focuses on an unrelated sentence. This happens because utilizing user’s feedback allows the model to understand a question better and locate the relevant input sentences. This illustrates the effectiveness of an interactive mechanism on addressing questions that require additional information. 2) The attention on both two problems can finally focus on the relevant sentences, showing the usefulness of our model for solving different types of QA problems.
6 CONCLUSION
In this paper, we present a self-adaptive model, CAN, which learns more accurate representations for statements and questions. More importantly, our model is aware what it knows and what it does not know within the context of the story, and takes an interactive mechanism to answer a question. Hence, our model takes an important step towards having a natural and intelligent conversation
with humans. In the future, we plan to employ more powerful attention mechanisms with explicit unknown state modeling and multi-round feedback-guided fine-tuning to make the model fully selfaware, self-adaptive, and self-taught. We also plan to expand our results to harder co-reference and interactive visual QA tasks with uncertainty modeling.
A SUPPLEMENTARY QUESTION ANALYSIS
We quantitatively evaluate the quality of supplementary question generated by our model based on IQA dataset. All baseline methods are designed to only predict an answer, none of them can generate a question. Thus we design another baseline method to generate supplementary question based on EncDec, and denote it as EncDec∗. Specifically, in training procedure, EncDec∗ takes statements and questions as input. If the supplementary question is available, it is used as output; otherwise the corresponding answer is viewed as output. Similar to our approach, the EOS symbol is used to determine whether the generated sentence is question or not, where the question ends with question mark and the answer ends with period symbol.
To test model’s ability to generate supplementary question, we define some following metrics. Suppose the number of problems is N , and the number of problems having supplementary question is Ns. Then Na = N −Ns is the number of remaining problems. Let SQueAcc = N̂sNs is the fraction of IQA problems which can be correctly estimated, and AnsAcc = N̂aNa is the fraction of remaining problems which can be correctly estimated as QA problem. So SQueAnsAcc = N̂s+N̂aN is the overall accuracy. In addition, the widely used BLEU (Papineni et al., 2002) and METEROR (Banerjee & Lavie, 2005) are also adopted to evaluate the quality of generated supplementary question. The results of our method and baseline method are presented in Table 9.
From the results, we can observe that 1) Two models can almost correctly determine whether it is time to output a question or not; 2) Two models are able to generate the correct supplementary questions whose contents exactly match with the ground truth. There is no surprise that EncDec∗ also performs well in generating question, because it is specifically designed for only outputting questions. Thus, if given enough training data, EncDec∗ could predict good questions. However, the limitation is that it cannot predict a supplementary question and an answer at the same time. Different from EncDec∗, our approach can accurately know when to output an answer or when to generate a supplementary question.

This paper proposes an "interactive" version of the bAbI dataset by adding supporting questions/answers to the dataset in cases where there is not enough information to answer the question. Interactive QA is certainly an interesting problem and is well-motivated by the paper. However, I don't feel like the bAbI extension is adequately explained. For example, the baseline DMN and MemN2N models on the IQA task are "take both statements and question as input and then
estimate an answer." Their task is then fundamentally more difficult from the CAN's because they do not distinguish "feedback" from the original context; perhaps a more fair approach would be to treat **every** question (both supporting and original questions) as individual instances. Also, how were the supporting questions and the user feedback generated? How many templates / words were used to create them? The dataset creation details are missing, and if space is an issue, a lot of basic exposition on things like GRU / sentence encodings can be cut (or at least greatly shortened) and replaced with pointers to the original papers.
Another issue I had is that the model attempts to generate these synthetic questions; if there are just one or two templates, why not just predict the values that fill these templates? So instead of generating "Which bedroom, master one or guest one?" with an RNN decoder, just predict "which" or "which bedroom"... isn't this sufficient? In the end, these just seem like more supporting facts, not actual interaction with users, and the fact that it is run on only three of the original twenty tasks make the conclusions hard to trust.
In conclusion, I think the paper has a strong idea and motivation, but the experiments are not convincing for the paper to be accepted at ICLR.


FRUSTRATINGLY SHORT ATTENTION SPANS IN NEURAL LANGUAGE MODELING
Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an attention mechanism over a differentiable memory have been proposed. For predicting the next token, these models query information from a memory of the recent history which can facilitate learning midand long-range dependencies. However, conventional attention mechanisms used in memoryaugmented neural language models produce a single output vector per time step. This vector is used both for predicting the next token as well as for the key and value of a differentiable memory of a token history. In this paper, we propose a neural language model with a key-value attention mechanism that outputs separate representations for the key and value of a differentiable memory, as well as for encoding the next-word distribution. This model outperforms existing memoryaugmented neural language models on two corpora. Yet, we found that our method mainly utilizes a memory of the five most recent output representations. This led to the unexpected main finding that a much simpler model based only on the concatenation of recent output representations from previous time steps is on par with more sophisticated memory-augmented neural language models.
1 INTRODUCTION
At the core of language models (LMs) is their ability to infer the next word given a context. This requires representing context-specific dependencies in a sequence across different time scales. On the one hand, classical N -gram language models capture relevant dependencies between words in short time distances explicitly, but suffer from data sparsity. Neural language models, on the other hand, maintain and update a dense vector representation over a sequence where time dependencies are captured implicitly (Mikolov et al., 2010). A recent extension of neural sequence models are attention mechanisms (Bahdanau et al., 2015), which can capture long-range connections more directly. However, we argue that applying such an attention mechanism directly to neural language models requires output vectors to fulfill several purposes at the same time: they need to (i) encode a distribution for predicting the next token, (ii) serve as a key to compute the attention vector, as well as (iii) encode relevant content to inform future predictions.
We hypothesize that such overloaded use of output representations makes training the model difficult and propose a modification to the attention mechanism which separates these functions explicitly, inspired by Miller et al. (2016); Ba et al. (2016); Reed & de Freitas (2015); Gulcehre et al. (2016). Specifically, at every time step our neural language model outputs three vectors. The first is used to encode the next-word distribution, the second serves as key, and the third as value for an attention mechanism. We term the model key-value-predict attention and show that it outperforms existing memory-augmented neural language models on the Children’s Book Test (CBT, Hill et al., 2016) and a new corpus of 7500 Wikipedia articles. However, we observed that this model pays attention mainly to the previous five memories. We thus also experimented with a much simpler model that only uses a concatenation of output vectors from the previous time steps for predicting the next token. This simple model is on par with more sophisticated memory-augmented neural language models. Thus, our main finding is that modeling short attention spans properly works well and provides notable
improvements over a neural language model with attention. Conversely, it seems to be notoriously hard to train neural language models to leverage long-range dependencies.
In this paper, we investigate various memory-augmented neural language models and compare them against previous architectures. Our contributions are threefold: (i) we propose a key-value attention mechanism that uses specific output representations for querying a sliding-window memory of previous token representations, (ii) we demonstrate that while this new architecture outperforms previous memory-augmented neural language models, it mainly utilizes a memory of the previous five representations, and finally (iii) based on this observation we experiment with a much simpler but effective model that uses the concatenation of three previous output representations to predict the next word.
2 METHODS
In the following, we discuss methods for extending neural language models with differentiable memory. We first present a standard attention mechanism for language modeling (§2.1). Subsequently, we introduce two methods for separating the usage of output vectors in the attention mechanism: (i) using a dedicated key and value (§2.2), and (ii) further separating the value into a memory value and a representation that encodes the next-word distribution (§2.3). Finally, we describe a very simple method that concatenates previous output representations for predicting the next token (§2.4).
2.1 ATTENTION FOR NEURAL LANGUAGE MODELING
Augmenting a neural language model with attention (Bahdanau et al., 2015) is straight-forward. We simply take the previous L output vectors as memory Yt = [ht−L · · · ht−1] ∈ Rk×L where k is the output dimension of a Long Short-Term Memory (LSTM) unit (Hochreiter & Schmidhuber, 1997). This memory could in principle contain all previous output representations, but for practical reasons we only keep a sliding window of the previous L outputs. Let ht ∈ Rk be the output representation at time step t and 1 ∈ RL be a vector of ones. The attention weights α ∈ RL are computed from a comparison of the current and previous LSTM outputs. Subsequently, the context vector rt ∈ Rk is calculated from a sum over previous output vectors weighted by their respective attention value. This can be formulated as
Mt = tanh(W Y Yt + (W hht)1 T ) ∈ Rk×L (1)
αt = softmax(wTMt) ∈ R1×L (2) rt = Ytα T ∈ Rk (3)
whereW Y ,W h ∈ Rk×k are trainable projection matrices andw ∈ Rk is a trainable vector. The final representation that encodes the next-word distribution is computed from a non-linear combination of the attention-weighted representation rt of previous outputs and the final output vector ht via
h∗t = tanh(W rrt +W xht) ∈ Rk (4)
where W r,W x ∈ Rk×k are trainable projection matrices. An overview of this architecture is depicted in Figure 1a. Lastly, the probablity distribution yt for the next word is represented by
yt = softmax(W ∗h∗t + b) ∈ R|V | (5)
whereW ∗ ∈ R|V |×k and b ∈ R|V | are a trainable projection matrix and bias, respectively.
2.2 KEY-VALUE ATTENTION
Inspired by Miller et al. (2016); Ba et al. (2016); Reed & de Freitas (2015); Gulcehre et al. (2016), we introduce a key-value attention model that separates output vectors into keys used for calculating the attention distribution αt, and a value part used for encoding the next-word distribution and context representation. This model is depicted in Figure 1b. Formally, we rewrite Equations 1-4 as follows:[
kt vt
] = ht ∈ R2k (6)
Mt = tanh(W Y [kt−L · · · kt−1] + (W hkt)1T ) ∈ Rk×L (7)
αt = softmax(wTMt) ∈ R1×L (8) rt = [vt−L · · · vt−1]αT ∈ Rk (9) h∗t = tanh(W rrt +W xvt) ∈ Rk (10)
In essence, Equation 7 compares the key at time step t with the previous L keys to calculate the attention distribution αt which is then used in Equation 9 to obtain a weighted context representation from values associated with these keys.
2.3 KEY-VALUE-PREDICT ATTENTION
Even with a key-value separation, a potential problem is that the same representation vt is still used both for encoding the probability distribution of the next word and for retrieval from the memory via the attention later. Thus, we experimented with another extension of this model where we further separate ht into a key, a value and a predict representation where the latter is only used for encoding the next-word distribution (see Figure 1c). To this end, equations 6 and 10 are replaced by[
kt vt pt
] = ht ∈ R3k (11)
h∗t = tanh(W rrt +W xpt) ∈ Rk (12)
More precisely, the output vector ht is divided into three equal parts: key, value and predict. In our implementation we simply split the output vector ht into kt, vt and pt. To this end the hidden dimension of the key-value-predict attention model needs to be a multiplicative of three. Consequently, the dimensions of kt, vt and pt are 100 for a hidden dimension of 300.
2.4 N -GRAM RECURRENT NEURAL NETWORK
Neural language models often work best in combination with traditional N -gram models (Mikolov et al., 2011; Chelba et al., 2013; Williams et al., 2015; Ji et al., 2016; Shazeer et al., 2015), since the former excel at generalization while the latter ensure memorization. In addition, from initial experiments with memory-augmented neural language models, we found that usually only the previous five output representations are utilized. This is in line with observations by Tran et al. (2016). Hence, we experiment with a much simpler architecture depicted in Figure 1d. Instead of an attention mechanism, the output representations from the previous N − 1 time steps are directly used to calculate next-word probabilities. Specifically, at every time step we split the LSTM output into N − 1 vectors [h1t , . . . ,hN−1t ] and replace Equation 4 with
h∗t = tanh WN  h 1 t ...
hN−1t−N+1
  ∈ Rk (13)
where WN ∈ Rk×(N−1)k is a trainable projection matrix. This model is related to higher-order RNNs (Soltani & Jiang, 2016) with the difference that we do not incorporate output vectors from the previous steps into the hidden state but only use them for predicting the next word. Furthermore, note that at time step t the first part of the output vector h1t will contribute to predicting the next word, the second part h2t will contribute to predicting the second word thereafter, and so on. As the output vectors from the N − 1 previous time-steps are used to score the next word, we call the resulting model an N -gram RNN.
3 RELATED WORK
Early attempts of using memory in neural networks have been undertaken by Taylor (1959) and Steinbuch & Piske (1963) by performing nearest-neighbor operations on input vectors and fitting parametric models to the retrieved sets. The dedicated use of external memory in neural architectures has more recently witnessed increased interest. Weston et al. (2015) introduced Memory Networks to explicitly segregate memory storage from the computation of the neural network, and Sukhbaatar et al. (2015) trained this model end-to-end with an attention-based memory addressing mechanism. The Neural Turing Machines by Graves et al. (2014) add an external differentiable memory with read-write functions to a controller recurrent neural network, and has shown promising results in simple sequence tasks such as copying and sorting. These models make use of external memory, whereas our model directly uses a short sequence from the history of tokens to dynamically populate an addressable memory.
In sequence modeling, RNNs such as LSTMs (Hochreiter & Schmidhuber, 1997) maintain an internal memory state as they process an input sequence. Attending over previous state outputs on top of an RNN encoder has improved performances in a wide range of tasks, including machine translation (Bahdanau et al., 2015), recognizing textual entailment (Rocktäschel et al., 2016), sentence summarization (Rush et al., 2015), image captioning (Xu et al., 2015) and speech recognition (Chorowski et al., 2015).
Recently, Cheng et al. (2016) proposed an architecture that modifies the standard LSTM by replacing the memory cell with a memory network (Weston et al., 2015). Another proposal for conditioning on previous output representations are Higher-order Recurrent Neural Networks (HORNNs, Soltani & Jiang, 2016). Soltani & Jiang found it useful to include information from multiple preceding RNN states when computing the next state. This previous work centers around preceding state vectors, whereas we investigate attention mechanisms on top of RNN outputs, i.e. the vectors used for predicting the next word. Furthermore, instead of pooling we use attention vectors to calculate a context representation of previous memories.
Yang et al. (2016) introduced a reference-aware neural language model where at every position a latent variable determines from which source a target token is generated, e.g., by copying entries from a table or referencing entities that were mentioned earlier.
Another class of models that include memory into sequence modeling are Recurrent Memory Networks (RMNs) (Tran et al., 2016). Here, a memory block accesses the most recent input words to selectively attend over relevant word representations from a global vocabulary. RMNs use a global memory with two input word vector look-up tables for the attention mechanism, and consequently have a large number of trainable parameters. Instead, we proposed models that need much fewer parameters by producing the vectors that will be attended over in the future, which can be seen as a memory that is dynamically populated by the language model.
Finally, the functional separation of look-up keys and memory content has been found useful for Memory Networks (Miller et al., 2016), Neural Programmer-Interpreters (Reed & de Freitas, 2015), Dynamic Neural Turing Machines (Gulcehre et al., 2016), and Fast Associative Memory (Ba et al., 2016). We apply and extend this principle to neural language models.
4 EXPERIMENTS
We evaluate models on two different corpora for language modeling. The first is a subset of the Wikipedia corpus.1 It consists of 7500 English Wikipedia articles (dump from 6 Feb 2015) belonging to one of the following categories: People, Cities, Countries, Universities, and Novels. We chose these categories as we expect articles in these categories to often contain references to previously mentioned entities. Subsequently, we split this corpus into a train, development, and test part, resulting in corpora of 22.5M words, 1.2M and 1.2M words, respectively. We map all numbers to a dedicated numerical symbol N and restrict the vocabulary to the 77K most frequent words, encompassing 97% of the training vocabulary. All other words are replaced by the UNK symbol. The average length of sentences is 25 tokens. In addition to this Wikipedia corpus, we also run experiments on the Children’s Book Test (CBT Hill et al., 2016). While this corpus is designed for cloze-style question-answering, in this paper we use it to test how well language models can exploit wider linguistic context.
4.1 TRAINING PROCEDURE
We use ADAM (Kingma & Ba, 2015) with an initial learning rate of 0.001 and a mini-batch size of 64 for optimization. Furthermore, we apply gradient clipping at a gradient norm of 5 (Pascanu et al., 2013). The bias of the LSTM’s forget gate is initialized to 1 (Jozefowicz et al., 2016), while other parameters are initialized uniformly from the range (−0.1, 0.1). Backpropagation Through Time (Rumelhart et al., 1985; Werbos, 1990) was used to train the network with 20 steps of unrolling. We reset the hidden states between articles for the Wikipedia corpus and between stories for CBT, respectively. We take the best configuration based on performance on the validation set and evaluate it on the test set.
5 RESULTS
In the first set of experiments we explore how well the proposed models and Tran et al.’s Recurrentmemory Model can make use of histories of varying lengths. Perplexity results for different attention window sizes on the Wikipedia corpus are summarized in Figure 2a. The average attention these models pay to specific positions in the history is illustrated in Figure 3. We observed that although our models attend over tokens further in the past more often than the Recurrent-memory Model, attending over a longer history does not significantly improve the perplexity of any attentive model.
The much simpler N -gram RNN model achieves comparable results (Figure 2b) and seems to work best with a history of the previous three output vectors (4-gram RNN). As a result, we choose the 4-gram model for the following N -gram RNN experiments.
1The wikipedia corpus is available at https://goo.gl/s8cyYa.
5.1 COMPARISON WITH STATE-OF-THE-ART MODELS
In the next set of experiments, we compared our proposed models against a variety of state-of-the-art models on the Wikipedia and CBT corpora. Results are shown in Figure 2c and 2d, respectively. Note that the models presented here do not achieve state-of-the-art on CBT as they are language models and not tailored towards cloze-sytle question answering. Thus, we merely use this corpus for comparing different neural language model architectures. We reimplemented the Recurrent-Memory model by Tran et al. (2016) with the temporal matrix and gating composition function (RM+tM-g).
Furthermore, we reimplemented Higher Order Recurrent Neural Networks (HORNNs) by Soltani & Jiang (2016).
To ensure a comparable number of parameters to a vanilla LSTM model, we adjusted the hidden size of all models to have roughly the same total number of model parameters. The attention window size N for the N -gram RNN model was set to 4 according to the best validation set perplexity on the Wikipedia corpus. Below we discuss the results in detail.
Attention By using a neural language model with an attention mechanism over a dynamically populated memory, we observed a 3.2 points lower perplexity over a vanilla LSTM on Wikipedia, but only notable differences for predicting verbs and prepositions in CBT. This indicates that incorporating mechanisms for querying previous output vectors is useful for neural language modeling.
Key-Value Decomposing the output vector into a key-value paired memory improves the perplexity by 7.0 points compared to a baseline LSTM, and by 1.9 points compared to the RM(+tM-g) model. Again, for CBT we see only small improvements.
Key-Value-Predict By further separating the output vector into a key, value and next-word prediction part, we get the lowest perplexity and gain 9.4 points over a baseline LSTM, a 4.3 points compared to RM(+tM-g), and 2.4 points compared to only splitting the output into a key and value. For CBT, we see an accuracy increase of 1.0 percentage points for verbs, and 1.7 for prepositions. As stated earlier, the performance of the Key-Value-Predict model does not improve significantly when increasing the attention window size. This leads to the conclusion that none of the attentive models investigated in this paper can utilize a large memory of previous token representations. Moreover, none of the presented methods differ significantly for predicting common nouns and named entities in CBT.
N -gram RNN Our main finding is that the simple modification of using output vectors from the previous time steps for the next-word prediction leads to perplexities that are on par with or better than more complicated neural language models with attention. Specifically, the 4-gram RNN achieves only slightly worse perplexities than the Key-Value-Predict architecture.
6 CONCLUSION
In this paper, we observed that using an attention mechanism for neural language modeling where we separate output vectors into a key, value and predict part outperform simpler attention mechanisms on a Wikipedia corpus and the Children Book Test (CBT, Hill et al., 2016). However, we found that all attentive neural language models mainly utilize a memory of only the most recent history and fail to exploit long-range dependencies. In fact, a much simpler N -gram RNN model, which only uses a concatenation of output representations from the previous three time steps, is on par with more sophisticated memory-augmented neural language models. Training neural language models that take long-range dependencies into account seems notoriously hard and needs further investigation. Thus, for future work we want to investigate ways to encourage attending over a longer history, for instance by forcing the model to ignore the local context and only allow attention over output representations further behind the local history.
ACKNOWLEDGMENTS
This work was supported by Microsoft Research and the Engineering and Physical Sciences Research Council through PhD Scholarship Programmes, an Allen Distinguished Investigator Award, and a Marie Curie Career Integration Award.

The paper presents an investigation of various neural language models designed to query context information from their recent history using an attention mechanism. The authors propose to separate the attended vectors into key, value and prediction parts. The results suggest that this helps performance. The authors also found that a simple model which which concatenates recent activation vectors performs at a similar level as the more complicated attention-based models.
The experimental methodology seems sound in general. I do have some issues with the way the dimensionality of the vectors involved in the attention-mechanism is chosen. While it’s good that the hidden layer sizes are adapted to ensure similar numbers of trainable parameters for all the models, this doesn’t control for the fact that key/value/prediction vectors of a higher dimensionality may simply work better regardless of whether their dimensions are dedicated to one particular task or used together. This separation clearly saves parameters but there could also be benefits of having some overlap of information assuming that vectors that lead to similar predictions may also be required in similar contexts for example. Some tasks may also require more dimensions than others and the explicit separation prevents the model from discovering and exploiting this.
While memory augmented RNNs and RNNs with attention mechanisms are not new, some of these architectures had not yet been applied to language modeling. Similarly (and as acknowledged by the authors), the strategy of separating key and value functionality has been proposed before, but not in the context of natural language modeling. I’m not sure about the novelty of the proposed n-gram RNN because I recall seeing similar architectures before but I understand that novelty was not the point of that architecture as it mainly serves as a proof of the lack of ability of the more complicated architectures to do better. In that sense I do consider it an inventive baseline that could be used in future work to test the ability of other models that claim to exploit long-term dependencies.
The exact computation of the representation h_t was initially not that clear to me (the terms hidden and output can be ambiguous at times) but besides this, the paper is quite clear and generally well-written.
The results in this paper are important because they show that learning long-term dependencies is not a solved problem by any means. The authors provide a very nice comparison to prior results and the fact that their n-gram RNN is often at least competitive with far more complicated approaches is a clear indication that some of those methods may not capture as much context information as previously thought. The success of the separation of key/value/prediction functionality in attention-based system is also noteworthy although I think this is something that needs to be investigated more thoroughly (i.e., with more control for hyperparameter choices).
Pros:
Impressive and also interesting results.
Good comparison with earlier work.
The n-gram RNN is an interesting baseline.
Cons:
The relation between the attention-mechanism type and the number of hidden units weakens the claim that the key/value/prediction separation is the reason for the increase in performance somewhat.
The model descriptions are not entirely clear.
I would have liked to have seen what happens when the attention is applied to a much larger context size.


RECURRENT HIDDEN SEMI-MARKOV MODEL
Segmentation and labeling of high dimensional time series data has wide applications in behavior understanding and medical diagnosis. Due to the difficulty of obtaining a large amount the label information, realizing this objective in an unsupervised way is highly desirable. Hidden Semi-Markov Model (HSMM) is a classical tool for this problem. However, existing HSMM and its variants typically make strong generative assumptions on the observations within each segment, thus their ability to capture the nonlinear and complex dynamics within each segment is limited. To address this limitation, we propose to incorporate the Recurrent Neural Network (RNN) as the generative process of each segment, resulting the Recurrent HSMM (R-HSMM). To accelerate the inference while preserving accuracy, we designed a structure encoding function to mimic the exact inference. By generalizing the penalty method to distribution space, we are able to train the model and the encoding function simultaneously. We also demonstrate that the R-HSMM significantly outperforms the previous state-of-the-art on both the synthetic and real-world datasets.
1 INTRODUCTION
Segmentation and labeling of time series data is an important problem in machine learning and signal processing. Given a sequence of observations {x1, x2, . . . , xT }, we want to divide the T observations into several segments and label each segment simultaneously, where each segment consists of consecutive observations. The supervised sequence segmentation or labeling techniques have been well studied in recent decades (Sutskever et al., 2014; Kong et al., 2015; Chen et al., 2015). However, for complicated signals, like human activity sensor data, accurately annotating the segmentation boundary or the activity type would be prohibitive. Therefore, it is urgent to develop unsupervised algorithms that can jointly learn segmentation and labeling information directly from the data without supervisions. Figure 1 provides an illustration which we are focus on.
The Hidden Semi-Markov Model (HSMM) (Murphy, 2002) is a powerful model for such task. It eliminates the implicit geometric duration distribution assumptions in HMM (Yu, 2010), thus allows the state to transit in a non-Markovian way. Most of the HSMM variants make strong parametric assumptions on the observation model (Rabiner, 1989; Johnson & Willsky, 2013; Yu, 2010). This makes the learning and inference simple, but ignores the nonlinear and long-range dependency within a segment. Take the human activity signals as an example. The movements a person performs at a certain time step would rely heavily on the previous movements, like the interleaving actions of left hand and right hand in swimming, or more complicated dependency like shooting after jumping in playing basketball. Some models have been proposed to tackle this problem (Ghahramani & Hinton, 2000; Fox et al., 2009; Linderman et al., 2016), but are limited in linear case.
Since people have justified RNN’s ability in modeling nonlinear and complicated dependencies (Sutskever et al., 2014; Du et al., 2016), we introduce the recurrent neural emission model into HSMM for capturing various dependencies within each segment to address such issue. However, the flexibility of recurrent neural model comes with prices: it makes the exact Expectation-Maximization (EM) algorithm computationally too expensive.
To speed up the learning and inference, we exploit the variational encoder (VAE) framework (Kingma & Welling, 2013). Specifically, we propose to use bidirectional RNN (bi-RNN) encoder. Such
architecture will mimic the forward-backward algorithm, and hence is expected to capture similar information as in exact posterior calculation.
It should be emphasized that due to the discrete nature of the latent variables in our model, the algorithm proposed in Kingma & Welling (2013) and its extension on time-series models (Gao et al., 2016; Krishnan et al., 2015) are not directly applicable. There are plenty of work proposed based on stochastic neuron (Tang & Salakhutdinov, 2013; Bengio et al., 2013; Mnih & Gregor, 2014; Raiko et al., 2014; Gu et al., 2015; Chung et al., 2016) to remedy such issue. However, none of these off-the-shelf methods are easy to achieve good performance according to our experiment: the hundreds or thousands layers of stochastic neuron (which is equal to the length of sequence), together with the switching generative RNN, make the encoding function very sensitive, and thus, extremely difficult to train fully on unsupervised setting. We propose a solution, stochastic distributional penalty method, which introduces auxiliary distributions to separate the decoding R-HSMM and encoding bi-RNN in training procedure, and thus, reduces the learning difficulty for each component. This novel algorithm is general enough and can be applied to other VAE with discrete latent variables, which can be of independent interest. We emphasize that the proposed algorithm is maximizing exact the nagative Helmholtz variational free energy. It is different from Johnson et al. (2016) in which a lower bound of the variational free energy is proposed as the surrogate to be maximized for convenience.
We experimentally justified our algorithm on the synthetic datasets and three real-world datasets, namely the segmentation tasks for human activity, fruit fly behavior and heart sound records. The R-HSMM with Viterbi exact inference significantly outperforms basic HSMM and its variants, demonstrating the generative model is indeed flexible. Moreover, the trained bi-RNN encoder also achieve similar state-of-the-art performances to the exact inference, but with 400 times faster inference speed, showing the proposed structured encoding function is able to mimic the exact inference efficiently.
2 MODEL ARCHITECTURE
Given a sequence x = [x1, x2, . . . , x|x|], where xt ∈ Rm is an m dimensional observation at time t, our goal is to divide the sequence into meaningful segments. Thus, each observation xt will have the corresponding label zt ∈ Z, where Z = {1, 2, . . . ,K} is a finite discrete label set and K is predefined. The label sequence z = [z1, z2, . . . , z|x|] should have the same length of x.
Besides labels, HSMM will associate each position twith additional variable dt ∈ D = {1, 2, . . . , D}, where dt is known as duration variable andD is the maximum possible duration. The duration variable can control the number of steps the current hidden state will remain. We use d to denote the duration sequence. We also use notation xt1:t2 to denote the substring [xt1 , xt1+1, . . . , xt2 ] of x. Without ambiguity, we use z as a segment label, and d as the duration.
In this paper, we focus on one of the variants of HSMM, namely the explicit duration HMM (EDHMM) (Rabiner, 1989), and use Decreasing Count Variables (Chiappa, 2014) for the notation.
Explicit Duration Hidden Markov Model. Similar to HMM, this model treats the pair of (z, d) as ‘macro hidden state’. The probability of initial macro state is defined as P (z, d) = P (z)P (d|z). We use the notation πz , P (z) and P (d|z) , Bz,d to parametrize the initial probability and duration probability, respectively. Ai,j , P (zt = i|zt−1 = j, dt−1 = 1) is the state transition probability on the segment boundary. Here π ∈ RK is in K-dimensional simplex. For each hidden state z, the corresponding rows Bz,: and Az,: are also in probability simplex. Here we assume the multinomial distribution for P (d|z). In EDHMM, the transition probability of macro hidden state P (zt, dt|zt−1, dt−1) is decomposed by P (zt|zt−1, dt−1)P (dt|zt, dt−1) and thus can be defined as:
P (zt|zt−1, dt−1) = { Azt−1,zt if dt−1 = 1 I(zt=zt−1) if dt−1 > 1 ; P (dt|zt, dt−1) = { Bzt,dt if dt−1 = 1 I(dt=dt−1−1) if dt−1 > 1 . (1)The graphical model is shown in Figure 2a.
Recurrent Hidden Semi-Markov Model. For the simplicity of explanation, we focus our algorithm on the single sequence first. It is straightforward to apply the algorithm for dataset that has multiple sequences. Given the parameters {π,A,B}, the log-likelihood of a single observation sequence x can be written as below,
L(x) = log ∑ z,d πz1Bz1,d1 |x|∏ t=2 P (zt|zt−1, dt−1)P (dt|zt, dt−1)P (x|z,d), (2)
where P (x|z,d) is the emission probability. To define P (x|z,d), we further denote the sequence variable s = [s1, s2, . . . , s|s|] to be the switching position (or the beginning) of segments. Thus s1 = 1, and si = si−1 + dsi−1 and |s| is the number of segments. Traditional HSMM assumes P (x|z,d) = ∏|x| t=1 P (xt|zt), which ignores the dependency and some degree of dynamics exhibited in each segment. While in this paper, we use RNN as the generative model to capture the dependent emission probability. Specifically, for the i-th segment starting from position si, the corresponding generative process is
P (xsi:si+dsi−1|zsi , dsi) = si+dsi−1∏ t=si P (xt|xsi:t−1, zsi) = si+dsi−1∏ t=si P (xt|ht, zsi) (3)
where we assume that the dependency of history before time step j can be captured by a vector hj ∈ Rh. As in RNN, we use a recurrent equation to formulate the history vector,
ht = σ(W (zsi )xt−1 + V (zsi )ht−1 + b (zsi )). (4) Finally, in this model, P (x|z,d) = ∏|s| i=1 P (xsi:si+dsi−1|zsi , dsi) is computed by the product of generative probabilities for each segment. In Eq. 4, W ∈ Rm×h is a weight matrix capturing the last observation xt−1, and V ∈ Rh×h is for the propagation of history ht−1. The b is a bias term. The superscript zsi indexes the RNN we used for the corresponding segment. The segments with different labels are generated using different RNNs. So we should maintain K RNNs. σ(·) is a nonlinear activation function. We use tanh in our experiments.
At the time step t, we assume a diagonal multivariate gaussian distribution over the conditional likelihood, where the mean and covariance matrix are the output of RNN, i.e.,
P (xt|ht, zsi) ∼ N (xt;µ = U (zsi ) µ ht + b (zsi ) µ ,Σ = Diag(exp(U (zsi ) Σ ht + b (zsi ) Σ ))) (5)
The matrices Uµ, UΣ ∈ Rm×h are used for parametrizing the mean and covariance at each time step j, given the history information. bµ, bΣ ∈ Rm are bias terms. For simplicity, let’s use θrnn = {θ(1)rnn, θ(2)rnn, . . . , θ(K)rnn} to denote the collection of parameters in each RNN. On the boundary case, i.e., the starting point of each segment, we simply set ht = 0, which can be viewed as the setting according to the prior knowledge (bias terms) of RNN.
The above formulation indicates that the generative model P (xt|ht, zsi) depends not only on the last step observation xt−1, but also the last hidden state ht−1, which is together captured in Eq. 4. In summary, we denote all the parameters in the proposed R-HSMM as θ = {π,A,B, θrnn}. The corresponding graphical model is shown in Figure 2b.
3 SEQUENTIAL VARIATIONAL AUTOENCODER
To obtain the posterior or MAP in the proposed R-HSMM, the classical forward-backward algorithm or Viterbi algorithm needs to solve one dynamic programming per sample, which makes the inference costly, especially for the long sequence with thousands of timestamps. So instead, we treat the Bayesian inference from optimization perspective, and obtain the posterior by maximizing the negative Helmholtz variational free energy (Williams, 1980; Zellner, 1988; Dai et al., 2016),
max Q(z,d|x)∈P
LθQ(x) := EQ(z,d|x) [logPθ(x, z,d)− logQ(z,d|x)] , (6)
over the space of all valid densities P . To make the optimization (6) tractable, the variational autoencoder restricts the feasible sets to be some parametrized density Qψ, which can be executed efficiently comparing to the forward-backward algorithm or Viterbi algorithm. However, such restriction will introduce extra approximation error. To reduce the approximation error, we use a structured model, i.e., bidirectional RNN, to mimic the dynamic programming in forward-backward algorithm. Specifically, in the forward-backward algorithm, the forward message αt(zt, dt) and backward message βt(zt, dt) can be computed recursively, and marginal posterior at position t depends on both αt(zt, dt) and βt(zt, dt). Similarly, in bi-RNN we embed the posterior message with RNN’s latent vector, and marginal posterior is obtained from the latent vectors of two RNNs at the same time step t. Let ψ = {ψ−−−→RNN1 , ψ←−−−RNN2 ,Wz ∈ R
h×K ,Wd ∈ Rh×D} be the parameters of bi-RNN encoder, the Qψ is decomposed as:
Qψ(z,d|x) = Q(z1|h1;ψ)Q(d1|z1, h1;ψ) |x|∏ t=2 Q(zt|dt−1, ht;ψ)Q(dt|zt, dt−1, ht;ψ), (7)
where ht = [ −−−→ RNN1(x1:t), ←−−− RNN2(xt:|x|)] is computed by bi-RNN. We use multinomial distributions Q(zt|ht;ψ) =M(softmax(W>z ht)) and Q(dt|zt, ht;ψ) =M(softmax(W>d ht)). The dependency over dt−1 ensures that the generated segmentation (z,d) is valid according to Eq. 1. For example, if we sampled duration dt−1 > 1 from Qψ at time t− 1, then dt and zt should be deterministic. In our experiment, we use LSTM (Hochreiter & Schmidhuber, 1997) as the recursive units in bi-RNN.
Since with any fixed θ, the negative Helmholtz variational free energy is indeed the lower bound of the marginal likelihood, i.e.,
logPθ(x) ≥ L(θ, ψ;x) := EQψ(z,d|x)[logPθ(x, z,d)− logQψ(z,d|x)], (8) therefore, we can treat it as a surrogate of the marginal log-likelihood and learn θ jointly with approximate inference, i.e.,
max θ,ψ
1
N N∑ n=1 L(θ, ψ;x(n)) (9)
It should be emphasized that due to the discrete nature of latent variables in our model, the algorithm proposed in Kingma & Welling (2013) is not directly applicable, and its extension with stochastic neuron reparametrization (Bengio et al., 2013; Raiko et al., 2014; Gu et al., 2015; Chung et al., 2016) cannot provide satisfied results for our model according to our experiments. Therefore, we extend the penalty method to distribution space to solve optimization (9).
Algorithm 1 Learning sequential VAE with stochastic distributional penalty method
1: Input: sequences {x(n)}Nn=1 2: Randomly initialize ψ(0) and θ0 = {π0, A0, B0, θ0rnn} 3: for λ = 0, . . . ,∞ do 4: for t = 0 to T do 5: Sample {x(n)}Mn=1 uniformly from dataset with mini-batch size M . 6: Get {z(n),d(n)}Mn=1 with θt by dynamic programming in (13). 7: Update πt+1, At+1, Bt+1 using rule (16). 8: θt+1rnn = θ t rnn,−γt 1M ∑M n=1∇θtrnnL̃λ(θ, ψ|x (n))
9: ψt+1 = ψt − ηt 1M ∑M n=1∇ψtL̃λ(θ, ψ|x(n)) . bi-rnn sequence to sequence learning
10: end for 11: end for
4 LEARNING VIA STOCHASTIC DISTRIBUTIONAL PENALTY METHOD
As we discussed, learning the sequential VAE with stochastic neuron reparametrization in unsupervised setting is extremely difficult, and none the off-the-shelf techniques can provide satisfied results. In this section, we introduce auxiliary distribution into (9) and generalize the penalty method Bertsekas (1999) to distribution space.
Specifically, we first introduce an auxiliary distribution Q̃(z,d|x) for each x and reformulate the optimization (9) as
max θ,ψ,{Q̃(z,d|x(n))}Nn=1
1
N N∑ n=1 EQ̃(z,d|x(n)) [ logPθ(x (n), z,d)− log Q̃(z,d|x(n)) ] , (10)
s.t. KL ( Q̃(z,d|x(n))||Qψ(z,d|x(n)) ) = 0, ∀x(n), n = 1, . . . , N.
We enforce the introduced Q̃(z,d|x) equals to Qψ(z,d|x) in term of KL-divergence, so that the optimization problems (9) and (10) are equivalent. Because of the non-negativity of KL-divergence, itself can be viewed as the penalty function, we arrive the alternative formulation of (10) as
max θ,ψ,{Q̃(z,d|x(n))}Nn=1
1
N N∑ n=1 L̃λ(θ, ψ|x(n)), (11)
where L̃λ(θ, ψ|x) = EQ̃(z,d|x) [ logPθ(x, z,d)− log Q̃(z,d|xi) ] − λKL ( Q̃(z,d|x)||Qψ(z,d|x) ) and λ ≥ 0. Obviously, as λ → ∞, KL(Q̃(z,d|x)||Qψ(z,d|x)) must be 0, otherwise the L̃∞(θ, ψ|x) will be −∞ for arbitrary θ, ψ. Therefore, the optimization (11) will be equivalent to problem (10). Following the penalty method, we can learn the model with λ increasing from 0 to ∞ gradually. The entire algorithm is described in Algorithm 1. Practically, we can set λ = {0,∞} and do no need the gradually incremental, while still achieve satisfied performance. For each fixed λ, we optimize Q̃ and the parameters θ, ψ alternatively. To handle the expectation in the optimization, we will exploit the stochastic gradient descent. The update rules for θ, ψ and Q̃ derived below.
4.1 UPDATING Q̃
In fact, fix λ, Qψ and Pθ in optimization (11), the optimal solution Q̃∗(z,d|x) for each x has closed-form. Theorem 1 Given fixed λ, Qψ and Pθ, Q̃∗(z,d|x) ∝ Qψ(z,d|x) λ 1+λPθ(x, z,d) 1 1+λ achieves the optimum in (11). Proof The proof is straightforward. Take the functional derivative of L̃ w.r.t. Q̃ and set it to zeros,
∇Q̃L̃ = logPθ(x, z,d) + λ logQ(z,d|x)− (1 + λ) log Q̃(z,d|x) = 0,
we obtain 11+λ logPθ(x, z,d) + λ 1+λ logQ(z,d|x) = log Q̃(z,d|x). Take exponential on both sides, we achieve the conclusion. In fact, because we are using the stochastic gradient for updating θ and ψ later, Q̃∗(z,d|x) is never explicitly computed and only samples from it are required. Recall the fact that Qψ(z,d|x) has a nice decomposition 7, we can multiply its factors into each recursion step and still get the same complexity
as original Viterbi algorithm for MAP or sampling. Specifically, let’s define αt(j, r) to be the best joint log probability of prefix x1:t and its corresponding segmentation which has the last segment with label j and duration r, i.e.,
αt(j, r) , max z1:t,d1:t log Q̃(z1:t,d1:t|x1:t), s.t. zt = j, dt = dt−r = 1, dt−r+1 = r (12)
here t ∈ {1, 2, . . . , |x|}, j ∈ Z, r ∈ D. Then we can recursively compute the entries in α as below:
αt(j, r) =  αt−1(j, r − 1) + 11+λ log( Bj,r Bj,r−1 P (xt|xt−r+1:t−1, z = j)) r > 1, t > 1 + λ1+λ log Qψ(dt−r+1=r|z=j,x) Qψ(dt−r+1=r−1|z=j,x) ; maxi∈Z\j maxr′∈D αt−1(i, r ′) + 11+λ log(Ai,jBj,1P (xt|z = j)) r = 1, t > 1 + λ1+λ logQψ(zt−r+1 = j, dt−r+1 = r|x); λ 1+λ logQψ(z1 = j, d1 = r|x) + 1
1+λ log(πjBj,1P (x1|z = j)); r = 1, t = 1 0. otherwise
(13) To construct the MAP solution, we also need to keep a back-tracing array βt(j, r) that records the transition path from αt−1(i, r′) to αt(j, r). The sampling from Q̃(z,d|x) also can be completed with almost the same style forwards filtering backwards sampling algorithm, except replacing the max-operator by sum-operator in α propagation Murphy (2012).
Without considering the complexity of computing emission probabilities, the dynamic programming needs time complexity O ( |x|K2 + |x|KD ) (Yu & Kobayashi, 2003) and O(|x|K) memory. We explain the details of optimizing the time and memory requirements in Appendix A.
Remark: When λ =∞, the Q̃(z,d|x) will be exactly Qψ(z,d|x) and the algorithm will reduce to directly working on Qψ(z,d|x) without the effect from Pθ(x, z,d). Therefore, it is equivalent to obtaining MAP or sampling of the latent variables z,d from Qψ(z,d|x), whose cost is O(|x|K). In practical, to further accelerate the computation, we can follow such strategy to generate samples when λ is already large enough, and thus, the effect of Pθ(x, z,d) is negligible.
4.2 UPDATING θ AND ψ
With the fixed Q̃(z,d|x), we can update the θ and ψ by exploiting stochastic gradient descent algorithm to avoid scanning the whole training set. Sample a mini-batch of sequences {xn}Mn=1 with size M N , we proceed to update {θ, ψ} by optimizing the Monte Carlo approximation of (11),
max θ,ψ
1
M M∑ n=1 logPθ(x (n), z(n),d(n)) + λ logQψ(z (n),d(n)|x(n)), (14)
where {z(n),d(n)} is the MAP or a sample of Q̃(z,d|x(n)). Note that the two parts related to θ and ψ are separated now, we can optimize them easily.
Update θ: Finding parameters to maximize the likelihood needs to solve the constrained optimization shown below
max θ
1
M M∑ n=1
( log π
z (n) 1
+ |s|∑ i=2 logA z (n) si−1 ,z (n) si + |s|∑ i=1 B z (n) si ,d (n) si + si+d (n) si −1∑
j=si
logP (x (n) j |h (n) j , z (n) si ; θrnn) ) (15)
where {π,A,B} are constrained to be valid probability distribution. We use stochastic gradient descent to update θrnn in totally K RNNs. For parameters π,A,B which are restricted to simplex, the stochastic gradient update will involve extra projection step. To avoid such operation which may be costly, we propose the closed-form update rule derived by Lagrangian,
πi =
∑M n=1 I(z (n) 1 = i)
m , Ai,j =
∑M n=1 ∑|s(n)|−1 t=1 I(z (n) st = i and z
(n) st+1 = j)∑M
n=1 |s(n)| −M (16)
Bj,r =
∑M n=1 ∑|s(n)| t=1 I(d (n) st = r and z
(n) st = j)∑M
n=1 |s(n)| Since we already have the segmentation solution, the total number of samples used for training is equal to the number of observations in dataset. The different RNNs use different parameters, and train on different parts of observations. This makes it easy for parallelized training.
Update ψ: Given fixed λ, logQψ(z(n),d(n)|x(n)) is essentially the sequence to sequence likelihood, where the input sequence is x and output sequence is {z,d}. Using the form of Qψ in Eq 7, this likelihood can be decomposed by positions. Thus we can conveniently train a bi-RNN which maximize the condition likelihood of latent variables by stochastic gradient descent.
Remark: We can get multiple samples {z,d} for each x from Q̃(z,d|x) to reduce the variance in stochastic gradient. In our algorithm, the samples of latent variable come naturally from the auxiliary distributions (which are integrated with penalty method), rather than the derivation from lower bound of objective (Tang & Salakhutdinov, 2013; Raiko et al., 2014; Mnih & Rezende, 2016).
5 EXPERIMENTS
Baselines We compare with classical HSMM and two popular HSMM variants. The first one is Hierarchical Dirichlet-Process HSMM (HDP-HSMM) (Johnson & Willsky, 2013), which is the nonparametric Bayesian extension to the traditional HSMM that allows infinite number of hidden states; the second one is called subHSMM (Johnson & Willsky, 2014), which uses infinite HMM as the emission model for each segment. This model also has two-level of latent structure. It considers the dependency within each segment, which is a stronger algorithm than HDP-HSMM. We also compare with the CRF autoencoder (CRF-AE) (Ammar et al., 2014), which uses markovian CRF as recognition model and conditional i.i.d.model for reconstruction. Comparing to HSMM, this model ignores the segmentation structures in modeling and is more similar to HMM. Evaluation Metric We evaluate the performance of each method via the labeling accuracy. Specifically, we compare the labels of each single observations in each testing sequence. Since the labels are unknown during training, we use KM algorithm (Munkres, 1957) to find the best mapping between predicted labels and ground-truth labels. Settings Without explicitly mentioned, we use leave-one-sequence-out protocol to evaluate the methods. Each time we test on one held-out sequence, and train on other sequences. We report the mean accuracy in Table 1. We set the truncation of max possible duration D to be 400 for all tasks. We also set the number of hidden states K to be the same as ground truth.
For the HDP-HSMM and subHSMM, the observation distributions are initialized as standard Multivariate Gaussian distributions. The duration is modeled by the Poisson distribution. We tune the concentration parameters α, γ ∈ {0.1, 1, 3, 6, 10}. The hyperparameters are learned automatically. For subHSMM, we tune the truncation threshold of the second level infinite HMM from {2 . . . 15}. For CRF-AE, we extend the origin model for the continuous observations, and learn all parameters similar to M. Schmidt (2008). We use mixture of Gaussians to model the emission, where the number of mixtures is tuned in {1, . . . , 10}. For the proposed R-HSMM, we use Adam (Kingma & Ba, 2014) to train the K generative RNN and bi-RNN encoder. To make the learning tractable for long sequences, we use back propagation through time (BPTT) with limited budget. We also tune the dimension of hidden vector in RNN, the L2-regularization weights and the stepsize. We implemented with CUDA that parallelized for different RNNs, and conduct experiments on K-20 enabled cluster. We include both the R-HSMM with the exact MAP via dynamic programming (rHSMM-dp) and sequential VAE with forward pass (rHSMM-fw) in experiments. In all tasks, the rHSMM-fw achieves almost the same performance to rHSMM-dp, but 400 times faster, showing the bi-RNN is able to mimic the forward-backward algorithm very well with efficient computation.
5.1 SEGMENTATION ACCURACY
Synthetic Experiments We first evaluate the proposed method on two 1D synthetic sequential data sets. The first data set is generated by a HSMM with 3 hidden states, where π,A,B are designed beforehand. A segment with hidden state z is a sine function λz sin(ωzx+ 1) + 2, where 1 and 2 are Gaussian random noises. Different hidden states use different scale parameters λz and frequency parameters ωz . The second data set also has 3 hidden states, where the segment with hidden state z is sampled from a Gaussian process (GP) with kernel function kz(x, y). Different hidden states employ different kernel functions. The specific kernel functions used here are k1(x, y) = exp{−min(| x−y | , |x+ y|)2/10}, k2(x, y) = exp{−(x− y)2/10} and k3(x, y) = (5− |x− y|)I{(5− |x− y|) < 5}. For both of the Sine and GP data sets, the duration of a segment is randomly sampled from a distribution defined on {1, ..., 100}, which depends on the hidden states. Thus, the segmentation task corresponds to finding out different functions embedded in the sequences.
HDP-HSMM 42.74 ± 2.73% 41.90 ± 1.58% 35.46 ± 6.19% 43.59 ± 1.58% 47.56 ± 4.31% 42.58 ± 1.54% CRF-AE 44.87 ± 1.63% 51.43 ± 2.14% 49.26 ± 10.63% 57.62 ± 0.22% 53.16 ± 4.78% 45.73 ± 0.66%
We visualize the segmentation results of ground truth and three competitors on Sine and GP data sets in Figure 1a and Figure 1b respectively, and report the numerical results in Table 1. As we can see, R-HSMM provides much better results on even small segments, dramatically outperforms HSMM variants and CRF-AE. Also note that, the sine function depicts short term dependencies, while Gaussian process has long dependency that determined by the kernel bandwidth. This demonstrates the ability of R-HSMM in capturing the long or short term dependencies.
Human activity This dataset which is collected by Reyes-Ortiz et al. (2016) consists of signals collected from waist-mounted smartphone with accelerometers and gyroscopes. Each of the volunteers is asked to perform a protocol of activities composed of 12 activities (see Figure 3a for the details). Since the signals within an activity type exhibit high correlation, it is natural for RNN to model this dependency. We use these 61 sequences, where each sequence has length around 3000. Each observation is a 6 dimensional vector, consists of triaxial measures from accelerometers and gyroscopes.
Figure 3a shows the ground truth and the segmentation results of all methods. Both rHSMM-dp and rHSMM-fw almost perfectly recover the true segmentation. They can also capture the transition activity types, e.g., stand to lie or sit to lie. The HSMM, HDP-HSMM and CRF-AE makes some fragmental but periodical segmentations for walking, caused by lacking the dependency modeling within a segment. The subHSMM also has similar problem, possibly due to the limited ability of HMM generative model.
Drosophila Here we study the behavior patterns of drosophilas. The data was collected by Kain et al. (2013) with two dyes, two cameras and some optics to track each leg of a spontaneously behaving fruit fly. The dimension of observation in each timestamp is 45, which consists of the raw features and some higher order features. See Figure 3b for the detail of the 11 behavior types. We perform leave-one-sequence-out experiment on 10 sequences of length 10000 each. Figure 3b shows the segmentation results on the prefix of one sequence, while Table 1 gives the mean accuracy on all sequences. Different from the previous experiment, where the human activity signals are relatively
smooth, here the signals depict high variance. Different activities exhibit quite different duration and patterns. Also, the activity types changes frequently. The R-HSMM almost captured each changing point of activities with both long and short durations. The corresponding mean accuracy also outperforms the baselines. However, we observed there are some correct segmentations with wrong labels. This happens mostly to the short segments, in which the RNN doesn’t have enough history established for distinguishing similar activity types.
Physionet The heart sound records, usually represented graphically by phonocardiogram (PCG), are key resources for pathology classification of patients. We collect data from PhysioNet Challenge 2016 (Springer et al., 2015), where each observation has been labeled with one of the four states, namely Diastole, S1, Systole and S2. We experiment with both the raw signals and the signals after feature extraction. Regarding the raw signals (Heart dataset), we collect 7 1-dimensional sequences of length around 40000. The feature-rich dataset (PN-Full) contains 2750 sequences, where each of them consists of 1500 4-dimensional observations. We do 5-fold cross validation for PN-Full. The visualization of segmentation results are shown in Appendix B.4. As the results shown in Table 1, our algorithm still outperforms the baselines significantly. Also for such long raw signal sequences, the speed advantage of bi-RNN encoder over Viterbi is more significant. Viterbi takes 8min to do one inference, while bi-RNN only takes several seconds. Our framework is also flexible to incorporate prior knowledge, like the regularity of heart state transition into HSMM.
5.2 RECONSTRUCTION
In this section, we examine the ability of learned generative model by visualizing the reconstructed signals. Given a sequence x, we use recognition model to get the latent variables z and d, then use learned K generative RNNs to generate signals within each segment. For the ease of visualization, we show the results on 1D signal dataset in Fig. 4a and Fig. 4b.
From Fig. 4 we can see the generative RNN correctly captures different characteristics from signals of different segment labels, such as different frequencies and scales in Sine dataset, or the different variance patterns in GP dataset. This is essential to distinguish between different segments.
6 CONCLUSION
We presented the R-HSMM, a generalization of HSMM by incorporating recurrent neural generative model as the emission probability. To eliminate the difficulty caused by such flexible and powerful model in inference, we introduced the bi-RNN as the encoding distribution via the variational autoencoder framework to mimic the forward-backward algorithm. To deal with the difficulty of training VAE containing discrete latent variables, we proposed a novel stochastic distributional penalty method. We justified the modeling power of the proposed R-HSMM via segmentation accuracy and reconstruction visualization. From the comprehensive comparison, the proposed model significantly outperforms the existing models. It should be emphasized that the structured bi-RNN encoder yields similar performance as the exact MAP inference, while being 400 times faster. Future work includes further speeding up of our algorithm, as well as generalizing our learning algorithm to other discrete variational autoencoder.
ACKNOWLEDGMENTS
This project was supported in part by NSF IIS-1218749, NIH BIGDATA 1R01GM108341, NSF CAREER IIS-1350983, NSF IIS-1639792 EAGER, ONR N00014-15-1-2340, Nvidia and Intel.
Appendix
A OPTIMIZING DYNAMIC PROGRAMMING
A.1 SQUEEZE THE MEMORY REQUIREMENT
In this section, we show that the Eq. 13 can be computed in a memory efficient way. Specifically, the dynamic programming procedure can be done with O(|x|K) memory requirement, and caching for precomputed emission probabilities requires O(D2K) memory space.
Update forward variable α Note that in Eq. 13, when r > 1, we can update αt(j, r) deterministically. So it is not necessary to keep the records for r > 1.
Specifically, let’s only record αt(j, 1), and do the updates in a similar way as in Eq. 13. The only difference is that, when constructing the answer, i.e., the last segment solution, we need to do a loop over all possible z and d in order to find the best overall segmentation solution.
It is easy to see that the memory consumption is O(|x|K).
Caching emission probability At each time step t, we compute P (xt+r|xt:t+r−1, z = j) for each j ∈ Z and r ∈ D. That is to say, we compute all the emission probabilities of observations starting from time t, and within max possible duration D. This can be done by performing feed-forward of K RNNs. After that, storing these results will require O(KD) space. For simplicity, we let etj,r = P (xt+r|xt:t+r−1, z = j), where et ∈ RK×D. Note that, at a certain time step t, we would require the emission probability of observations P (xt|xt−r+1:t−1, z = j) for some j ∈ Z and r ∈ D. In this case, the corresponding first observation is xt−r. That is to say, we should keep et−D+1, . . . , et at time step t. This makes the memory consumption goes to O(KD2)
A.2 SQUEEZE THE TIME COMPLEXITY
In Eq. 13, the most expensive part is when r = 1 and t > 1. If we solve this in a naive way, then this step would require O(|x|K2D) for time complexity, which is quite expensive. Here we adopt similar technique as in Yu & Kobayashi (2003). Let γt(i) = maxr′∈D αt−1(i, r′), then we can get
αt(j, r) = max i∈Z max r′∈D
αt1(i, r ′) +
1
1 + λ log(Ai,jBj,1P (xt|z = j)) (17)
+ λ
1 + λ logQψ(zt−r+1 = j, dt−r+1 = r|x)
= max i∈Z
γt−1(i) + 1
1 + λ log(Ai,jBj,1P (xt|z = j)) (18)
+ λ
1 + λ logQψ(zt−r+1 = j, dt−r+1 = r|x)
This reduces the complexity to be O(|x|K2).
B MORE EXPERIMENT RESULTS
B.1 SYNTHETIC DATASETS
The reconstructed signals from the original signals are shown in Fig. 5 and Fig. 6 for sine dataset and gaussian Process dataset respectively. We can see the reconstructed signal almost recovered the original signal. The RNN captured the key differences of states, such as the frequency and scale; while in gaussian process dataset, it also recovered the complicated pattern involving long term dependencies.
We show the confusion matrix of all methods on synthetic sine and gaussian process dataset in Figure 7 and Figure 8 respectively.
B.2 HUMAN ACTIVITY
The confusion matrices of our method and two baseline algorithms are shown in Figure 9.
In Figure 10, we also show several other segmentation results on different testing sequences.
B.3 DROSOPHILA
The confusion matrices of our method and two baseline algorithms are shown in Figure 11.
Since each sequence is too long to be clearly shown in one figure, we split the segmentation results of one sequence into four parts, and show them in Figure 12.
B.4 HEART SOUND RECORDS
The confusion matrices of our method and two baseline algorithms are shown in Figure 13.
Also, we split the segmentation results of one sequence into four parts, and show them in Figure 14.

This paper presents a novel model for unsupervised segmentation and classification of time series data.  A recurrent hidden semi-markov model is proposed.  This extends regular hidden semi-markov models to include a recurrent neural network (RNN) for observations.  Each latent class has its own RNN for modeling observations for that category.  Further, an efficient training procedure based on a variational approximation.  Experiments demonstrate the effectiveness of the approach for modeling synthetic and real time series data.
This is an interesting and novel paper.  The proposed method is a well-motivated combination of duration modeling HMMs with state of the art observation models based on RNNs.  The combination alleviates shortcomings of standard HSMM variants in terms of the simplicity of the emission probability.  The method is technically sound and demonstrated to be effective.
It would be interesting to see how this method compares quantitatively against CRF-based methods (e.g. Ammar, Dyer, and Smith NIPS 2014).  CRFs can model more complex data likelihoods, though as noted in the response phase there are still limitations.  Regardless, I think the merits of using RNNs for the class-specific generative models are clear.