Recent advances in Question Answering have lead to the development of very complex models which compute rich representations for query and documents by capturing all pairwise interactions between query and document words. This makes these models expensive in space and time, and in practice one has to restrict the length of the documents that can be fed to these models. Such models have also been recently employed for the task of predicting dialog responses from available background documents (e.g., Holl-E dataset). However, here the documents are longer, thereby rendering these complex models infeasible except in select restricted settings. In order to overcome this, we use standard simple models which do not capture all pairwise interactions, but learn to emulate certain characteristics of a complex teacher network. Specifically, we first investigate the conicity of representations learned by a complex model and observe that it is significantly lower than that of simpler models. Based on this insight, we modify the simple architecture to mimic this characteristic. We go further by using knowledge distillation approaches, where the simple model acts as a student and learns to match the output from the complex teacher network. We experiment with the Holl-E dialog data set and show that by mimicking characteristics and matching outputs from a teacher, even a simple network can give improved performance. The advent of large scale datasets for QA has lead to the development of increasing complex neural models with specialized components for (i) encoding the query (ii) encoding the document(s) (iii) capturing interactions between document and query words and (iv) generating/extracting the correct answer span from the given document (Seo et al., 2016; Hu et al., 2017; Yu et al., 2018). While these models give state-of-the-art performance on a variety of datasets, they have very high space and time complexity. This is a concern, and in practice, it is often the case that one has to resort to restricting the maximum length of the input document such that the model can run with reasonable resources (say, a single 12GB Tesla K80 GPU). Such complex span prediction models are also being adapted for other NLP tasks such as dialog response prediction (Moghe et al., 2018), which is the focus of this work. In particular, we refer to the Holl-E dataset where the task is to extract the next response from a document which is relevant to the conversation (see Figure 1). This setup is very similar to QA wherein the input is { context, document } as opposed to { query, document } and the correct response span needs to be extracted from the given document. Given this similarity, it is natural to adopt existing QA models (Seo et al., 2016; Yu et al., 2018) for this task. However, the documents in Holl-E dataset are longer, and the authors specifically report that they were unable to run these models when the entire document was given as input. Hence, they report results only in constrained oracle settings where the document is trimmed such that the response still lies in the shortened document. The above situation suggests that there is clearly a trade-off needed. On one hand, we want to harness the power of these complex models to achieve better performance and on the other hand we want to be able to run them with reasonable compute resources without arbitrarily trimming the input document. This trade-off situation naturally leads to the following question: Is it possible to build a simple model, with low memory and compute requirements, that copy desirable characteristics from complex models? To answer this, we start with a relatively simple model with very basic components for encoding query, document and capturing interactions. Once these interactions are cap-Source Doc: ...comes in. As soon as the door is open, the Bride's fist crashes into Vernita's face. A savage fight follows, first with fists, then with knives.... The fight pauses ... At this point Vernita is introduced as a member of the Deadly Vipers, codename Copperhead. ... Sample Conversation: Prober (S1): Which is your favourite character in this? Responder (S2): My favorite character was Copperhead because she was kicking butt. Prober (S3): Oh my goodness I agree, because the fight with Vernita was the best in the whole movie. Responder (S4): It's starts off action packed because as soon as the door is open, the Bride's fist crashes into Vernita's face. A savage fight follows, first with fists, then with knives. Prober (S5): And it gets better when we find out they are both assassins. Responder (S6): And a group of them, Vernita is introduced as a member of the Deadly Vipers, codename Copperhead. tured, the model computes a final representation which is then fed to a decoder to predict the correct span in the document. This recipe is very similar to BiDAF (Seo et al., 2016), QANeT (Yu et al., 2018) but the main difference is that these models use much more complex encoder and interaction components to arrive at the final representation. As expected, the performance of this model is poor when compared to BiDAF, QANeT. The aim now is to improve the performance of this model by carefully analysing or learning from complex models. Given that the complex model differs in the manner in which the final representation is computed, one hypothesis is that it learns richer final representations than the simple model. Indeed, on investigation, we found that the final representations learned by complex models are diverse for different inputs (context, document pairs) as compared to the simple model. Based on this insight, we propose a modification to the simple model which increases the diversity of the embeddings, thereby improving the performance. While this insight obtained by manual investigation is useful, there is clearly scope for learning by exploring other characteristics of the model. One principled way of doing this is to use knowledge distillation (Hinton et al., 2015) where the simple model acts as a student and learns to mimic the probability distributions predicted by a teacher. In other words, instead of simply maximizing the log likelihood of the training data, the simple model now gets additional signals from the teacher which act as hints while training. Our experiments, using the Holl-E dataset show that by (i) improving the conicity (Chandrahas et al., 2018) of the representations learned by the simple model and (ii) mimicking the outputs of the complex teacher model the simple model can give improved performance with fewer compute and memory requirements. In particular, when compared to a standalone simple model the student model shows an improvement of 3 . 4 % (compare SAM-mul-train (LD) and SAM-add-topk (LD) entries in Table 2 and Table 3 respectively). Over the past few years neural sequence prediction models which take a question as input and predict the corresponding answer span in a given document have evolved rapidly. Such models have also been adapted for dialog response prediction in the context of the Holl-E dataset (Moghe et al., 2018). These models typically differ in the components used for capturing interactions between query and document, capturing interactions between sentences in a document and refin-ing the query/document representation over multiple passes (Shen et al., 2017; Dhingra et al., 2017; Sordoni et al., 2016). In particular, a co-attention network which computes the importance of every query word w.r.t. every document word and the importance of every document word w.r.t. every query word is an important component in most state of the art models (Hermann et al., 2015; Kadlec et al., 2016; Cao et al., 2016; Xiong et al., 2016; Seo et al., 2016; Gong and Bowman, 2017; Dhingra et al., 2017; Wang et al., 2017; Shen et al., 2017; Trischler et al., 2016; Group and Asia, 2017; Tan et al., 2017; Sordoni et al., 2016). Similarly, some models (Group and Asia, 2017; Seo et al., 2016; Hu et al., 2017) contain a self-attention network which computes the importance of every document word w.r.t. every other document word. In general, the most successful models (for example, BiDAF (Seo et al., 2016), QANeT (Yu et al., 2018)) use a combination of these components which capture all pairwise interactions and are thus computationally very expensive. As a result, in practice, these models are not suitable for longer documents. compact models (Cheng et al., 2017). For example, Ba and Caruana (2014); Hinton et al. (2015); Lopez-Paz et al. (2016); Chen et al. (2017) train a shallow student network using soft targets (or class probabilities) generated by an expensive teacher instead of the hard targets present in the training data. Romero et al. (2015) extend this idea to train a student model using the intermediate representations learned by the teacher model which act as additional hints . This idea of Knowledge Distillation has also been tried in the context of pruning networks for multiple object detection (Chen et al., 2017), speech recognition (Wong and Gales, 2016). In the context of reading comprehension or span prediction, Hu et al. (2018) have very recently shown that we can distill knowledge from an ensemble of models into a single model. However, unlike our work, the single model itself is a complex model (Hu et al., 2017) containing an expensive self attention network and a RL agent. To the best of our knowledge, ours is the first work which tries to build a simple span prediction model by distilling knowledge from a complex model. We view a conversation as sequence of utterances by a prober and a responder . The response prediction (RP) model aims to predict the utterance by the responder based on a source document, when given the query (prober's most recent utterance) and the history (past utterance by the prober and responder). See Figure 1 for an example. We denote the lengths of source document, query, prober history and responder history as T, I, J, K . The LSTMs/GRUs used all have the same number of cells, denoted by d . In particular, the document length T is of the order of a few thousands and the query/history lengths I, J, K are of the order of a few hundreds. Contrast this with QA tasks, where T is only of the order of a few hundreds, and the query length ( I + J + K ) is of the order of a few tens. BiDAF (Seo et al., 2016) is an extremely popular model used for span prediction in reading comprehension based question answering problems. We can frame the problem of response prediction as one of question answering by concatenating the query, prober history, and responder history into a single question. BiDAF has proven to be hugely successful in QA tasks, but has a large number of parameters (about 2.5 million) and consumes a large amount of computational space and time during training and prediction. We use the BiDAF model as a guiding post while constructing our model, and in particular focus on the so called query to context attention , which is a vector (denoted by (cid:101) h ) that indicates the weighted sum of the most important words in the source document, with respect to the query and histories. QANeT (Yu et al., 2018) is another recent model used for span prediction in QA tasks and specifically targets better space and time efficiency than BiDAF. Despite this, it still has a large number of parameters (about 1.3 million) and still consumes a large amount of computational space and time during training and prediction. The QANeT model can also be modified for response prediction in a similar manner to BiDAF. We now describe the simple attention model that we aim to learn. In a fashion similar to that of BiDAF and QANeT architectures, the simple model also operates in 3 distinct layers. See Figure 2 for an overview into the model. The words from the source document, the utterances by the prober and the responder are all encoded using standard GloVe embeddings (Pen-nington et al., 2014). In the next layer we encode the query (prober's most recent utterance) using a BiGRU/BiLSTM, and encode the previous utterances of the prober and responder in a query sensitive manner. Query Encoder: Embedded query words are passed through BiGRU where final state q I  R 2 d acts as query representation. Query Sensitive History Summariser: The history of the prober and responder are passed through a BiGRU to get context sensitive vectors h P j  R 2 d and h R k  R 2 d for j  [ J ] and k  [ K ] . These vectors are combined to get vectors h P and h R . This process of combining uses the query representation q I , and hence h P and h R can Query to Prober history attention Query to Responder history attention q I is Which this? be viewed as query-aware representations of the prober and responder history. The equations for h P are given below. The vector h R is also calculated in a similar manner. e j = mul WP ( h P j , q I )  = softmax( e ) h P = (cid:88) j  j h P j where mul W ( v 0 , v 1 ) = v (cid:62) 0 W v 1 is a parameterized multiplicative way of capturing the interaction between two vectors. The source document is finally used in this layer to predict the start and end indices of the response. The GloVe embedded words of the source document are passed through a BiGRU to get context sensitive vectors u t  R 2 d , for all t  [ T ] . Each index t gets a score s t based on the interaction between u t and the query/history vectors q I , h P , h R . The scores s t are normalized by a softmax and is taken to be the prediction of the starting word index. where mul W ( u 0 , . . . , u a ) = u (cid:62) 0 (cid:0)(cid:80) ai =1 W i u i (cid:1) A similar method is used for the prediction the ending word index as well. We performed several experiments on the Holl-E dataset and observed that the complex models (QANeT and BiDAF) perform better than the simple attention model described in Section 3.3. However, they take significantly more time and memory for training and inference. In fact, for the examples with longer source documents, both BiDAF and QANeT run into memory issues when training. During prediction, the memory issues in QANeT and BiDAF can be sidestepped by breaking the source document into multiple chunks and taking the highest scoring span. In the rest of this section we study several approaches to nudge the simple attention model to take parameters that make it have similar behaviour as the complex models, and check if the so nudged model demonstrates better performance on the Holl-E dataset. same conversation are often the same, even though the right response is different in those points. For example, consider the conversation in Figure 1. We expect the trained model to be such that Pred span ( SD, S 1 , S 2 , S 3 ) = span SD ( S 4 ) where SD is the source document, and S i are the the utterances in the conversation. Similarly, we expect the trained model to be such that Pred span ( SD, S 3 , S 4 , S 5 ) = span SD ( S 6 ) However we often find that our simple model predicts the same span for both the cases above, which is wrong (unless S 4 and S 6 are the same.) We hypothesize this as being due to the context sensitive embeddings of the history not depending strongly on the query, and hence the span prediction model picks up most information from the source document. To support this point of view we measured the diversity of the context-to-query vectors (cid:101) h of the BiDAF model for several examples grouped by conversation. In more detail we computed the conicity (Chandrahas et al., 2018) of vectors (cid:101) h ( SD, S 1 , S 2 , S 3 ) , (cid:101) h ( SD, S 3 , S 4 , S 5 ) , . . . for every conversation in the test set and averaged it over all conversations. (See Figure 3 for an overview on conicity). This average conicity was observed to be about 0 . 6 (see Table 6), which, according to Chandrahas et al. (2018), is low (low conicity implies high diversity). We observe similar behaviour for QANeT as well. The average conicity of the row sums of the similarity matrix grouped by conversation was also observed to be about 0 . 6 (see Table 6). On the other hand, for our simple attention model, the average conicity of the vectors h R and h P , when computed in a similar fashion as mentioned above were generally high (about 0 . 8 ) (see Table 6). Based on these observations we hypothesize that decreasing the conicity of the vectors h R and h P would improve the performance of the simple attention model. In particular, we propose to change the multiplicative method of combining vectors into an additive method instead. In particular we propose to replace the function mul in our simple model with the function add defined as follows: add W ( v 0 , v 1 , . . . , v a ) = w (cid:62) tanh (cid:32) a (cid:88) i =0 W i v i (cid:33) where the vector w and the matrices W i parameterize the mode of combining the input vectors. This is motivated by Chandrahas et al. (2018) who show that using additive model in embedding of entities in knowledge graphs gives consistently better diversity than using multiplicative models. While borrowing high level ideas from complex models, like increasing diversity of the learned representation can help to some extent, one can push this further to distill the learned complex model (Hinton et al., 2015) into the simple attention model. To achieve this, we train a teacher model (BiDAF or QANeT) on the training set and use it to make predictions on the same training set. The simple attention model would minimise the sum of two loss functions: 1) Cross entropy loss of the predicted start and end indices with the train labels of the start and end indices, 2) KL-divergence of the predicted start and end indices from the teacher prediction of the same. The loss on a single training sample is given below D ( p Tb || p S b ) + D ( y b || p S b ) + D ( p Te || p Se ) + D ( y e || p Se ) (1) where D denotes the KL divergence, p Tb , p Te denote the predicted begin index and end index distribution of the teacher model, and p Sb , p Se denote the predicted begin and end index distribution of the student model and y b , y S denote the true begin and end index in one-hot vector form. Another variant of knowledge distillation is as follows. We do not view the teacher predicted distribution for all indices with importance and just take the top few predicted indices. In particular the loss on a single training sample is given below D ( (cid:101) p Tb || (cid:101) p Sb ) + D ( y b || p Sb ) + D ( (cid:101) p Te || (cid:101) p Se ) + D ( y e || p Se ) (2) where (cid:101) p Tb , (cid:101) p Te gives just the (normalised) probability of the topk predictions of teacher model on the begin and end indices. Similarly (cid:101) p Sb , (cid:101) p Se gives the student predictions for the begin and end indices restricted to the topk entries given by the teacher model. As the teacher model is already trained, and the main objective in knowledge distillation is to have the student model mimic the teacher model, there is no need to restrict the objective terms 1 and 3 in equation 1 to only the training data. Hence by hallucinating conversations and documents we can get more terms in the objective and has an effect similar to data augmentation. Another possible way to take advantage of teacher models is to extract more information than simply the predicted spans for each training example from the teacher models. In particular one easy way to extract piece of information is the gradient of the model output with respect to the input for the teacher model. The so called Sobolev training (Czarnecki et al., 2017) exploits this information and adds two more extra terms to the objective in (1). The gradients are all taken with respect to the model input, which would be the source document, the query and the histories. In this section, we describe the setup used for our experiments and discuss the results. We perform experiments using the Holl-E conversation dataset (Moghe et al., 2018) which contains crowdsourced conversations from the movie do-main. Every conversation in this dataset is associated with background knowledge comprising of plot details (from Wikipedia), reviews and comments (from Reddit). Every alternate utterance in the conversation is generated by copying and/or modifying sentences from this unstructured background knowledge. We refer the reader again to Figure 1 for a sample from this dataset. We use the same train, test and validation splits as provided by the authors of the original paper (Moghe et al., 2018). For each chat in the training data, the authors construct training triplets of the form { document, context, response } where the number of train, test and validations triplets are 34486, 4388 and 4318 respectively. The context contains (i) the query (the prober's most recent utterance) and (ii) the history (past 2 utterances by the prober and the responder) as described earlier. The task then is to train a model which can predict the response given the document and the context. At test time, the model is shown document, context and predicts the response. As mentioned earlier, the authors of Holl-E found that BiDAF and QANeT run into memory issues when evaluated on their dataset. Hence, they propose two setups (i) long document (LD) setup and (ii) short document (SD) setup. In the long document setup, the authors do not trim the document from which the response needs to be predicted. In the short document setup, the authors trim the document to 256 words such that the span containing the response is contained in the trimmed document. This enables them to evaluate BiDAF and QANeT on the trimmed document. We also report experiments using both the LD and SD setup. As mentioned above complex models (BiDAF and QANeT) face memory issues on training set with long documents. So for all situations where we need predictions from complex models for long documents, we use a BiDAF/QANeT model trained on short document examples, and the prediction on the long document is made by splitting the long documents into chunks and feeding it to the trained BiDAF/QANeT model. The final predicted span is the largest scoring span across all chunks. For all models, we considered the following hy-perparameters and tuned them using the validation set. We tried batch sizes of 32 and 64 and the following GRU sizes: 64, 100, 128. We experimented with 1, 2 and 3 layers of GRU. We used pre-trained publicly available Glove word embeddings 1 of 100 dimensions. The best performance 1 https://nlp.stanford.edu/projects/ glove/ SAM, SD SAM, LD BiDAF, SD BiDAF, LD QANeT, SD QANeT, LD Memory 540MB 1.3GB 11GB 11GB 3GB 3GB Time 30 secs 43 sec 347 secs 710 secs 90 secs 150 secs Table 1: Inference' Memory and Time usage for different models. was with the batch size of 32, 2 layers of GRU with hidden size 64. We used Adam (Kingma and Ba, 2014) optimizer with initial learning rate set to 0 . 001 ,  1 = 0 . 9 ,  2 = 0 . 999 . We performed L2 weight decay with decay rate set to 0 . 001 . The models that we experiment with are listed below: with additive interactions and no teacher terms in the objective (Only terms 2 and 4 in Eqn. (1)). 2. SAM-add-Teach : The simple attention model with additive interactions and only knowledge distillation terms in the objective (Only terms 1 and 3 in Eqn. (1)). 3. SAM-add : The simple attention model with additive interactions and both knowledge distillation terms and training data terms in the objective (all terms in 1). 4. SAM-add-topk : The simple attention model with additive interactions and knowledge distillation applied to the topk indices and training data terms in the objective (all terms in 2). 5. SAM-add-aug : The SAM-add model, where the teacher terms are evaluated on hallucinated data in addition to training data. The hallucinated data are derived from the original training set by reordering the words in the source document, query and histories. 6. SAM-add-grad : The SAM-add model, with extra terms in the loss penalising the deviation of the gradient of the simple model from the gradient of the teacher model. as the SAM-add model, but has 6 terms instead of the 4 terms in Equation 1. The extra two terms arise from using both QANeT and BiDAF instead of just one. 8. SAM-add-ensemble : Same as the SAM-add model, but the teacher predictions p T are set as the average of the QANeT and BiDAF predictions. All the add models above also have a mul variant where the additive interaction add is replaced by a multiplicative interaction mul . The F1-scores of the various models we train are given in Table 2, Table 3 and Table 4. A summary of the space and time complexity of prediction with the simple model and the complex models is given in Table 1. The training times and parameter counts of the models are given in Table 5 We draw several conclusions and inferences from these results and make some comments below. Efficient Training with Simple Model: From Table 5, we observe that simple attention model has 5 to 10 times less parameters than QANeT and BiDAF. The training time of the simple model is also significantly lesser than that of the complex models. Efficient Prediction with Simple Model: From Table 1, we observe that the simple model takes significantly less memory and time during prediction as well. The complex models run out of memory on the large document test set, but a prediction can still be made with a trained BiDAF or QANeT Model Details BiDAF QANeT SAM-add-teach 20.23 SAM-mul 40.81 40.76 SAM-add 42.05 41.89 SAM-add-topk 41.71 42.01 SAM-add-aug 41.65 41.62 SAM-add-ensemble 41.74 SAM-add-both 42.32 SAM-add-grad 41.37 41.72 Table 4: F1 Scores for different variants of simple attention model on short document test set. Conicity of Multiplicative models vs Additive Models: As noted before we use two distinct methods to capture the interaction between a group of vectors : an additive mechanism add , and a multiplicative mechanism mul . As mentioned earlier, an additive model for capturing interactions has been hypothesized to increase diversity. This is true in our case as well: the conicity of the h P and h R vectors goes down from about 0 . 8 to 0 . 7 (compare SAM-mul and SAM-add entries in Table 6) when using the additive model instead of the multiplicative model. F1 scores of Multiplicative models vs Additive Models: In addition to improving diversity, using the additive model add instead of the multiplicative model mul increases F1-scores all across the board. We have not reported scores for certain multiplicative variants because their performance is significantly worse. F1 scores of Simple Model with Knowledge Distillation: We observe that using a teacher model for knowledge distillation using the objective in (1) almost always improves the performance of the simple model. Importance of training labels: The objective in knowledge distillation (Equation (1)) involves both the training labels and the teacher predicted distribution. Even though the teacher predicted distribution also incorporates the training data, removing the training data term from the objective of knowledge distillation worsens performance significantly. Topk Distillation: The knowledge distillation approach based on topk predicted indices results in the best simple model for long document examples (see Table 3). The value of k was chosen to be 50 for the short document case and it is 20 for the long document case. Add-Both Knowledge Distillation: Learning from multiple teachers could lead to better performance hence we trained the student model with two teachers (BiDAF+QANeT). Here the objective function of student is to minimize KL Divergence between predictions for both teachers. We achieved best results with this technique on short document (SD) test set. Data Augmentation and Gradient Distillation: While the data augmentation and gradient distillation methods hold a lot of promise, in the experiments that we conducted, we did not see a signifi-cant improvement. QANeT Teachers vs BiDAF Teachers: Using either QANeT or BiDAF as a teacher doesn't seem to make any difference in the performance of the student models (compare the two columns in Table 3 and 4). In this work, we address the trade-off between simple models on one hand which have low memory and compute requirements and complex models on the other hand which give better performance but are computationally expensive. We propose a middle ground by training a simple model to mimic the characteristics of a complex model. In particular, we make observations from a complex model which learns very diverse representations for different inputs and suitably modify the simple model to learn similar diverse representations. We go further, by using knowledge distillation techniques to improve the simple model by training it to match the outputs from the complex model. We experimented with the Holl-E conversation dataset and showed that by mimicking characteristics of the teacher a simple model can give improved performance. We thank Department of Computer Science and Engineering, and Robert Bosch Center for Data Sciences and Artificial Intelligence, IIT Madras (RBC-DSAI) for providing us with adequate compute resources. Lastly, we thank Ananya Sai and Shweta Bhardwaj for valuable discussions and reviewing intial drafts of this paper.
abstain abstain abstain abstain method objective method method result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain method abstain abstain abstain abstain abstain abstain objective objective objective objective objective result abstain abstain abstain abstain abstain abstain result objective abstain abstain abstain result abstain abstain abstain abstain other other other other other other other other other other other abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method other abstain abstain abstain abstain abstain abstain abstain result method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method method method other abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective method result result other other