Recent end-to-end task oriented dialog systems use memory architectures to incorporate external knowledge in their dialogs. Current work makes simplifying assumptions about the structure of the knowledge base (such as the use of triples to represent knowledge) and combines dialog utterances (context), as well as, knowledge base (KB) results, as part of the same memory. This causes an explosion in the memory size, and makes reasoning over memory, harder. In addition, such a memory design forces hierarchical properties of the data to be fit into a triple structure of memory. This requires the memory reader to learn how to infer relationships across otherwise connected attributes. In this paper we relax the strong assumptions made by existing architectures and use separate memories for modeling dialog context and KB results. Instead of using triples to store KB results, we introduce a novel multilevel memory architecture consisting of cells for each query and their corresponding results. The multi-level memory first addresses queries, followed by results and finally each key-value pair within a result. We conduct detailed experiments on three publicly available task oriented dialog data sets and we find that our method conclusively outperforms current state-of-the-art models. We report a 15-25% increase in both entity F1 and BLEU scores. Task oriented dialog systems are designed to complete a user specified goal, or service an information request using natural language exchanges. Unlike open domain end-to-end neural dialog models, task oriented systems rely on external knowledge sources, outside of the current conversation context, to return a response (Henderson  Work done during internship at IBM Research AI et al., 2014a; Su et al., 2016; Bordes and Weston, 2017a; Eric and Manning, 2017; El Asri et al., 2017). For instance, in the example shown in Table 1, a dialog agent giving tour package recommendations needs to be able to first query an external knowledge source to determine packages that meet a user's requirement, and then respond accordingly. In order to enable end-to-end goal oriented dialog tasks, current state of the art methods use neural memory architectures to incorporate external knowledge (Su et al., 2016; Eric and Manning, 2017; Madotto et al., 2018). As can be seen in Table 1, agent responses may also include entity values present only in the dialog context (eg: Mu-nich in the Agent response in Turn 2). In order to support such utterances, models also include tokens from the input dialog context in the same memory (Madotto et al., 2018). Existing memory based architectures for task oriented dialog suffer from multiple limitations. First, the creation of a shared memory for copying values from dialog context, as well as the knowledge base (KB) results, forces the use of a common memory reader for two different types of data. This makes the task of reasoning over memory, harder  not only does the memory reader need to determine the right entries from a large memory (since each word from context also occupies a memory cell), it also needs to learn to distinguish between the two forms of data (context words and KB results) stored in the same memory. Second, all current neural memory architectures store results, returned by a knowledge source, in the form of triples (eg. subject  relation  object ). This modeling choice makes it hard for the memory reader to infer relationships across otherwise connected attributes. For instance, consider the example triple store in Table 2 showing results for a query executed for packages between Dallas and Mannheim. If the user asks the dialog agent to check the price of stay at a 5 star hotel, the memory reader needs to infer that the correct answer is $2800 by learning that the price, category and hotel need to be linked inorder to return an answer (shown in blue). Lastly, current models treat conversations as a sequential process, involving the use of KB results from only the most recent information re-quest/query. In contrast, in real world dialogs such as the one shown in Table 1, the agent may have to refer to results ( to Mannheim ) from a previously executed query (see Turn 7). Thus, at each turn, the system has to memorize all the information exchanged during the dialog, and infer the package being referred to, by the user. In order to support such dialogs, the memory needs to store results of all queries executed during the course of the dialog. The problem of inferring over such results (which may be from multiple queries) is exacerbated when memory is represented in the form of triples. In this paper, we present our novel multi-level memory architecture that overcomes the limitations of existing methods: (i) We separate the memory used to store tokens from the input context and the results from the knowledge base. Thus, we learn different memory readers for context words as well for knowledge base entities (ii) Instead of using a subj  rel  obj store, we develop a novel multi-level memory architecture which encodes the natural hierarchy exhibited in knowledge base results by storing queries and their corresponding results and values at each level. We first attend on the queries, followed by the results in each query to identify the result being referred to, by the user. We then attend on the individual entries in the result to determine which value to copy in the response. Figure 1c shows our multi-level memory storing the results from queries executed as part of the dialog in Table 1. Our paper makes the following contributions: 1. We propose the use of separate memories for copying values from context and KB results. Thus, the model learns separate memory readers for each type of data. 2. Our novel multi-level memory for KB results, models the queries, results and their values in their natural hierarchy. As our experiments show, the separation of memory as well as our multi-level memory architecture, both, contribute to signifi-cant performance improvements. 3. We present detailed experiments demonstrating the benefit of our memory architecture along with model ablation studies. Our experiments on three publicly available datasets ( CamRest676 (Su et al., 2016), InCar assistant (Eric and Manning, 2017), Maluuba Frames (El Asri et al., 2017)) show a substantial improvement of 15-25 % in both entity F1 scores, and BLEU scores as compared to existing state of the art architectures. To the best of our knowledge, we are the first to attempt end-to-end modeling of task oriented dialogs with non-sequential references as well as multiple queries, as seen in the Maluuba Frames dataset. A human evaluation on model outputs also shows our model is preferred by users over existing systems such as KVRet (Eric and Manning, 2017) and Mem2Seq (Madotto et al., 2018). Recent methods, such as (Vinyals and Le, 2015; Serban et al., 2016, 2017), proposed for end-to-end learning of dialogs were aimed at modeling open-domain dialogs. While they can be used for learning task oriented dialogs, they are not well suited to interface with a structured KB. To better adapt them to handle task oriented dialogs: 1) (Bordes and Weston, 2017b) proposed a memory network based architecture to better encode KB tuples and perform inferencing over them and 2) (Madotto et al., 2018) incorporated copy mechanism to enable copying of words from the past utterances and words from KB while generating responses. All successful end-to-end task oriented dialog networks (Eric and Manning, 2017; Bordes and Weston, 2017b; Madotto et al., 2018) make assumptions while designing the architecture: 1) KB results are assumed to be a triple store, 2) KB triples and past utterances are forced to be represented in a shared memory to enable copying over them. Both these assumptions make the task of inferencing much harder. Any two fields linked directly in the KB tuple are now linked indirectly by the subject of the triples. Further, placing the KB results and the past utterances in same memory forces the architecture to encode them using a single strategy. In contrast, our work uses two different memories for past utterances and KB results. The decoder is equipped with the ability to copy from both memories, while generating the response. The KB results are represented using a multi-level memory which better reflects the natural hierarchy encoded by sets of queries and their corresponding result sets. Memory architectures have also been found to be helpful in other tasks such as question answering. Work such as (Xu et al., 2016) defines a hierarchical memory architecture consisting of sentence level memory followed by word memory for a QA task while (Chandar et al., 2016) defines a memory structure that speeds up loading and inferencing over large knowledge bases. Recent work by (Chen et al., 2018) uses a variational memory block along with a hierarchical encoder to improve diversity of open domain dialog responses. In this section, we describe our end-to-end model for task oriented dialogues. Our model 1 (Figure 1a) consists of: (i) a hierarchical encoder which encodes the current input context consisting of the user and agent utterances (ii) a multi-level memory that maintains the queries and knowledge base re-1 Code is available at Multi-Level Memory sults seen so far in the course of the dialogue, and (iii) copy augmented sequence decoder that uses separate context and multi-level memory. The queries and their corresponding results are maintained in a multi-level memory. The decoder uses a gating mechanism for memory selection while generating a response. Our model uses a standard hierarchical encoder as proposed by (Sordoni et al., 2015). The encoder takes a sequence of utterances as input. For the t th turn, the dialogue context can be represented as ( c 1 , c 2 , ...c 2 t  1 ) , which consists of t user utterances and t  1 system utterances. Each utterance c i is further a sequence of words ( w i 1 , w i 2 , ...w im ) . We first embed each word w ij using a word embedding function  emb that maps each word to a fixed-dimensional vector. We then generate utterance representations,  ( c i ) using a single layer bi-directional GRU. h eij denotes the hidden state of word w ij in the bidirectional GRU. The input representation c is generated by passing each utterance representation  ( c i ) through another single layer GRU. Motivation: Current approaches break down KB results by flattening them into ( subj-rel-obj ) triples. However, converting KB results into triples leads to loss of relationship amongst attributes in the result set. This makes the reasoning over memory difficult as model now has to infer relationships when retrieving values from memory. Instead, we use a multi-level memory which keeps the natural hierarchy of results intact (without breaking them into triples). We store the queries and their corresponding results and individual values at different levels . We first attend on the queries and then on the results for each query to identify which result the user is referring to. This also enables us to handle user requests that refer to results from a previously executed query. We propose that a representation of all the values in the result, and not just one of the values (desig-nated as subj ), should be used while attending over a result in KB. We attend on this compound representation of the result before attending on the individual key-value pairs in each result, to determine which value to copy into the generated response. attention Table 1 Figure 1: Model architecture (a) along with schematic representation of context memory (b) and multi-level KB memory Let q 1 , q 2 , ...q k be the queries fired to the knowledge base till the current state of the dialogue. Every query q i is a set of key-value pairs { k q i a : v q i a , 1 < a < n q i } , corresponding to the query's slot and argument where n q i is the number of slots in query q i . For example, after the user utterance at Turn 3 in Table 1, the query fired by the system on the knowledge base would be { 'origin':'Dallas','destination':'Manheim','Start': 'Aug 26', 'end': 'Aug 31', 'Adults':1 } . The execution of a query on an external knowledge base, returns a set of results. Let r ij be the j th result of query q i . Each result r ij is also a set of slot-value pairs { k r ij a : v r ij a , 1 < a < n r ij } where n r ij is the number of attributes in result r ij . A visualization of the memory with queries and their corresponding results can be seen in Figure 1c. The first level of memory contains the query representations. Each query q i is represented by q vi = Bag of words over the word embeddings of values ( v q i a ) in q i . The second level of memory contains the result representations. Representation of each result r ij is given by r vij = Bag of words over the word embeddings of values ( v r ij a ) in r ij . The third level of memory contains the result cells which have the key-value pairs ( k r ij a : v r ij a ) of the results. The values ( v r ij a ) which are to be copied into the system response are thus present in the final level of memory . We now describe how we apply attention over the context and multi-level memory. The model generates the agent response word-by-word; a word at time step t is either generated from the decode vocabulary or is a value copied from one of the two memories (knowledge base or context memory). A soft gate g 1 controls whether a value is generated from vocabulary or copied from memory. Another gate g 2 determines which of the two memories is used to copy values. Let the hidden state of the decoder at time t be h t The hidden state h t is used to apply attention over the input context memory. Attention is applied over the hidden states of the input bidirectional (BiDi) GRU encoder using the  con-cat  scheme as given in (Luong et al., 2015). The attention for the j th word in the i th utterance is given by: a ij = exp ( w T 1 tanh ( W 2 tanh ( W 3 [ h t , h eij ]))) (cid:80) ij exp ( w T 1 tanh ( W 2 tanh ( W 3 [ h t , h eij ]))) (2) The attention scores a ij are combined to create an attended context representation d t , d t = (cid:88) i,j a i,j h eij (3) and similar to (Luong et al., 2015), the decoder word-generation distribution is given by : P g ( y t ) = softmax ( W 1 [ h t , d t ] + b 1 ) (4) 3.3.2 Copying words from context memory: The input context memory is represented using the hidden states h eij of the input Bi-Di GRU encoder. Similar to (Gulcehre et al., 2016), the attention scores a ij , are used as the probability scores to form the copy distribution P con ( y t ) over the input context memory. The context representation d t , along with the hidden state of decoder h t , is used to attend over the multi-level memory. The first level attention,  . , is applied over the queries q . . The second level attention,  i. , is the attention over the results r i. of query q i . The product of first level attention and second level attention is the attention over results of all the queries in the multi-level memory. The weighted sum of the first level attention, second level attention and result representations gives us the attended memory representation, m t . Each result is further composed of multiple result cells. On the last level of memory, which contains the result cells, we apply key-value attention similar to (Eric and Manning, 2017). The key of the result cell is the word embedding of the slot, k r ij a , in the result. The attention scores,  ij. , for the keys represent the attention over the result cells of each result r ij . The product of first level attention  i , second level attention  ij and third level attention  ijl gives the final attention score of the value v r ij l in the KB memory. These final attention scores when combined (Eq. 10), form the copy distribution, P kb ( y t ) , over the values in KB memory. Similar to (Gulcehre et al., 2016), we combine the generate and copy distributions we use gate g 2 (Eq. 11) to obtain the copy distribution P c ( y t ) (Eq. 12) by combining P kb ( y t ) and P con ( y t ) . Finally, we use gate g 1 to obtain the final output distribution P ( y t ) , by combining generate distribution P g ( y t ) and copy distribution P c ( y t ) as shown below: We present our experiments using three real world publicly available multi-turn task oriented dialogue datasets: the InCar assistant (Eric and Manning, 2017), CamRest (Su et al., 2016) and the Maluuba Frames dataset (El Asri et al., 2017). All three datasets contain human-human task oriented dialogues which were collected in a Wizard-of-Oz (Wen et al., 2017) setting. (i) InCar assistant dataset consists of 3031 multi-turn dialogues in three distinct domains: calendar scheduling, weather information retrieval, and point-of-interest navigation. Each dialogue has it's own KB information provided and thus, the system does not have to make any queries. (ii) CamRest dataset , consists of 676 human-to-human dialogues set in the restaurant reservation domain. There are three queryable slots (food, price range, area) that users can specify. This dataset has currently been used for evaluating slot-tracking systems. Recent work by (Lei et al., 2018) uses an end-to-end network without a KB and substitutes slot values with placeholders bearing the slot names in agent responses. However, we formatted the data to evaluate end-to-end systems by adding API call generation from the slot values so that restaurant suggestion task can proceed from the KB results. (iii) Maluuba Frames dataset , consists of 1369 dialogues developed to study the role of memory in task oriented dialogue systems. The dataset is set in the domain of booking travel packages which involves flights and hotels. In contrast to the previous two datasets, this dataset contains dialogs that require the agent to remember all information presented previously as well as support results from multiple queries to the knowledge base. A user's preferences may change as the dialogue proceeds, and can also refer to previously presented queries (non-sequential dialog). Thus, to store multiple queries, we require 3 levels in our multi-level memory as compared to 2 levels in the other datasets, since they don't have more than one query. We do not use the dialogue frame annotations and use only the raw text of the dialogues. We map ground-truth queries to API calls that are also required to be generated by the model. Recent work has used this dataset only for frame tracking (Schulz et al., 2017) and dialogue act prediction (Peng et al., 2017; Tang et al., 2018). To the best of our knowledge we are the first to attempt the end-to-end dialog task using this dataset. Table 3 summarizes the statistics of the datasets. In this section, we briefly describe how the knowledge base queries are generated as API calls as part of the model response. The InCar assistant dataset has a fixed KB for each dialogue whereas the CamRest and Maluuba datasets require queries to be fired on a global KB. Queries in CamRest dataset can have 3 slots namely cuisine, area and pricerange, whereas those in Maluuba can have 8 slots, which are destination, origin, start date, end date, budget, duration, number of adults and children. Any query that is to be fired on the KB is expected to be generated by the model as an API call, by considering a fixed ordering of slots in the generated response. For eg., in CamRest dataset, ApiCall(area=south, pricerange=cheap) would be generated by the model as api call dontcare south cheap , with dontcare meaning that the user does not have any preference for cuisine and, south , cheap being the user constraints for area and pricerange respectively. Therefore, the task of API call generation typically involves copying relevant entities that are present in dialog context. Our model is trained end-to-end using Adam optimizer (Kingma and Ba, 2014) with a learning rate of 2 . 5 e  4 . The batch-size is sampled from [8,16]. We use pre-trained Glove vectors (Pen-nington et al., 2014) with an embedding size of 200. The GRU hidden sizes are sampled from [128, 256]. We tuned the hyper-parameters with grid search over the validation set and selected the model which gives best entity F1. We use the commonly used BLEU metric (Pap-ineni et al., 2002) to study the performance of our systems as it has been found to have strong correlation (Sharma et al., 2017) with human judgments in task-oriented dialogs. To explicitly study the behaviour of different memory architectures, we use the entity F 1 to measure how effectively values from the knowledge base are used in the dialog. To compute the entity F1, we micro-average the precision and recall over the entire set of system responses to compute the micro F1 2 . For the InCar Assistant dataset, we compute a per-domain entity F1 as well as the aggregated entity F1. Since our model does not have slot-tracking by design, we evaluate 2 We observe that (Madotto et al., 2018) reports the micro average of recall as the micro F1. We experiment with the following baseline models for comparing the performance of our Multi-Level Memory architecture: Ptr-UNK 3 (Gulcehre et al., 2016): The model augments a sequence-to-sequence architecture with attention-based copy mechanism over the encoder context. KVRet (Eric and Manning, 2017): The model uses key value knowledge base in which the KB is represented as triples in the form of subject  relation  object . This model does not support copying words from context. The sum of word embeddings of subject , relation is used as the key of the corresponding object . Mem2Seq 3 (Madotto et al., 2018): The model uses a memory networks based approach for attending over dialog history and KB triples. During decoding, at each time step, the hidden state of the decoder is used to perform multiple hops over a single memory which contains both dialog history and the KB triples to get the pointer distribution used for generating the response. Table 4 shows the performance of our model against our baselines. We find that our multilevel memory architecture comprehensively beats all existing models, thereby establishing new state-of-theart benchmarks on all three datasets. Our model outperforms each baseline on both BLEU and entity F1 metrics. InCar: On this dataset, we show entity F1 scores for each of the scheduling, weather and navigation domains. Our model has the highest F1 scores across all the domains. It can be seen that our 3 We use the implementation provided by (Madotto et al., 2018) at https://github.com/HLTCHKUST/Mem2Seq model strongly outperforms Mem2Seq on each domain. A detailed study reveals that the use of triples cannot handle cases when a user queries with non-subject entries or in cases when the response requires inferencing over multiple entries. In contrast, our model is able to handle such cases since we use a compound representation of entire result (bag of words over values) while attending on that result. CamRest: Our model achieves the highest BLEU and entity F1 scores on this dataset. From Table 4, we see that simpler baselines like Ptr-UNK show competitive performance on this dataset because, as shown in Table 3, CamRest dataset has relatively fewer KB entries. Thus, a simple mechanism for copying from context results in good entity F1 scores. Maluuba Frames: The Maluuba Frames dataset was introduced for the frame tracking task. Here, a dialog frame is a structured representation of the current dialog state. Instead of explicitly modeling the dialog frames, we use the context representation d t to directly attend on the Multi-level memory. As Table 3 shows, this dataset contains significantly longer contexts as well as larger number of entities, as compared to the other two datasets. In addition, unlike other datasets, it also contains non-linear dialog flows where a user may refer to previously executed queries and results. The complexity of this dataset is reflected in the relatively lower BLEU and F1 scores as compared to other datasets. To further understand the effect of separating context memory from KB memory and using a multi-InCar level memory for KB, Table 5 shows the percentage of ground-truth entities, according to their category, which were also present in the generated response. For example, on the InCar dataset, out of the 930 entities in ground-truth response that were to be copied from the KB, our model was able to copy 37 . 5% of them into the generated response. From Table 5, it can be seen that our model is able to correctly copy a significantly larger number of entities from both, KB and context, as compared to the recent Mem2Seq model in all datasets. We report results from ablation studies on all three datasets. Table 6 shows the incremental benefit obtained from individual components used in our model. We investigate the gains made by (i) Using separate memory for context and KB triples (ii) Replacing KB triples with a Multi-level memory. We use the recent Mem2Seq model for comparison with a unified context and KB memory model. As can be seen from Table 6, the separation of context memory and KB memory leads to a significant improvement in BLEU and F1 scores on all datasets. This validates our hypothesis that storing context words and KB results in a single memory confuses the memory reader. The use of a multi-level memory instead of triples leads to further gains. This suggests, better organization of KB result memory by keeping the natural hierarchy intact is beneficial. We analyzed the errors made by our dialog model on 100 dialog samples in test set of Maluuba Frames. We observed that the errors can be divided into five major classes: (i) Model outputs wrong KB result entry due to incorrect attention (27%), (ii) Model returns package details instead of asking for more information from the user (16%), (iii) Model incorrectly captures user intent (13%), (iv) Model makes an error due to nonsequential nature of dialog (22%). In such errors, our model either generates an API call for a result already present in memory, or our model asks for a query-slot value that was already provided by the user, (v) Data specific characteristics such as in-sufficient samples for certain classes of utterances (eg: more than one package returned) or returning different, but meaningful package attributes as compared to ground-truth data, contribute to 22% of the errors. We also conducted a blind user study that compared outputs from our model, Mem2Seq and KVRet systems. We used 96 randomly selected examples from each test split of Maluuba and CamRest datasets resulting in a total of 192 examples. Our study was split across 8 users who were provided with results fetched from the KB, current dialog context, gold response and the outputs of each of the models. Model outputs were shuffled in each example and users were asked to score each output between 1 (lowest) to 5 (highest) in terms of its accuracy of information in response and the quality of language. The results of this study are presented in Table 7. We also report the MRR (mean-reciprocal rank) for model preference along with other scores. It can be seen that our model consistently ranks high for both information accuracy and language quality as well as reports a higher MRR. To further understand the quality of model performance, we asked the human evaluators whether their best ranked model output was a useful response. We saw that the evaluators agreed in 76.04% and 58.33% of the cases for CamRest and Maluuba datasets respectively. We observe that the results from human evaluation go hand-in-hand with automatic evaluation and reinforce our claim that separating context, KB memory and using a multilevel representation for the KB memory are useful for improving dialog modeling. Role Turn Utterance Agent 1 hello! howcanihelpyoutoday? Analyzing the attention weights is a useful way to understand how the model is inferencing over the memory to copy entities from it. Table 8 shows an example of a dialog from the Maluuba Frames dataset and the outputs generated by different models. Here, the user first wants to know about packages to Manas' and then requests for trips to Pittsburgh'. Later, the user becomes interested in the 3.5 star hotel in Pittsburgh which was suggested by the agent and wants to know its guest rating. It can be seen from Table 8 that our model outputs the correct guest rating (8.86) of the hotel. Mem2Seq fails to understand the context and generates an irrelevant response. KVRet generates a readable response but points to the guest rating of a different hotel. (b) Decreasing order of attention scores over words in dialogue context Figure 2: Visualization of attention over memory while generating the word 8.86' for the example in Table 8. The attention over the memory while generating the word  8 . 86 ' for this example is shown in Fig 2. Fig 2a shows that the query with destination as Pittsburgh' gets the highest attention and among the results of this query, the package with the 3.5 star rated hotel gets highest attention. Within this result, the model gives highest score to the result cell with guest rating as the key. To further understand why the correct result hotel gets higher attention, Fig 2b shows the attention scores over the words in context memory. The context representation d t captures the important words (3.5, guest, rating) in context which are in-turn used to apply attention over the multi-level memory. Lastly, studying the values of the gates g 1 (prob. of generating from vocabulary) and g 2 (prob. of copying from KB), we found that gate g 1 had a probability value of 0.08 thereby driving the model to copy from memory instead of generating from output vocabulary and gate g 2 , with a probability value of 0.99 , was responsible for selecting KB memory over context memory. In this paper, we presented an end-to-end trainable novel architecture with multi-level memory for task oriented dialogues. Our model separates the context and KB memory and combines the attention on them using a gating mechanism. The multi-level KB memory reflects the natural hierarchy present in KB results. This also allows our model to support non-sequential dialogs where a user may refer to a previously suggested result. We find that our model beats existing approaches by 15-25% on both entity F1 and BLEU scores, establishing state-of-the-art results on three publicly available real-world task oriented dialogue datasets. In a user study comparing outputs from our system against recent models, we found that our model consistently scored higher for both language quality as well as correctness of information in the response. We also present the ben-efits of each of our design choices by performing an ablation study. In future work, we would like to incorporate better modeling of latent dialog frames so as to improve the attention signal on our multi-level memory. As our error analysis suggests, nearly 22% of the errors could possibly be reduced by improved modeling of the dialog context. We believe that model performance can also be improved by capturing user intent better in case of non-sequential dialog flow.
abstain abstain abstain abstain abstain result objective abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result abstain objective objective result objective abstain objective objective objective result objective result other other other other other other other other objective other other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain other abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result result abstain result result abstain result result abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain result objective method abstain result result result method result method result