The Knowledge Base (KB) used for real-world applications, such as booking a movie or restaurant reservation, keeps changing over time. End-to-end neural networks trained for these task-oriented dialogs are expected to be immune to any changes in the KB. However, existing approaches breakdown when asked to handle such changes. We propose an encoder-decoder architecture (BOSSNET ) with a novel Bag-of-Sequences (BOSS ) memory, which facilitates the disentangled learning of the re-sponse's language model and its knowledge incorporation. Consequently, the KB can be modified with new knowledge without a drop in interpretability. We find that BOSSNET outperforms state-of-the-art models, with considerable improvements ( > 10%) on bAbI OOV test sets and other human-human datasets. We also systematically modify existing datasets to measure disentanglement and show BOSSNET to be robust to KB modifications. Task-oriented dialog agents converse with a user with the goal of accomplishing a specific task and often interact with a knowledge-base (KB). For example, a restaurant reservation agent (Henderson et al., 2014) will be grounded to a KB that contains the names of restaurants, and their details. In real-world applications, the KB information could change over time. For example, (1) a KB associated with a movie ticket booking system gets updated every week based on new film releases, and (2) a restaurant reservation agent, trained with the knowledge of eateries in one city, may be deployed in other cities with an entirely different range of establishments. In such situations, the system should have the ability to conform to new-found knowledge unseen during its training. Ideally, the training algorithm must learn to disentangle the language  D. Raghu is an employee at IBM Research. model from the knowledge interface model. This separation will enable the system to generalize to KB modifications, without a loss in performance. Moreover, for achieving good progress towards the user's task, the agent must also retain the ability to draw inferences based on past utterances and the KB. Notably, we find that existing approaches either achieve this disentanglement or effective progress towards the task, but not both. For instance, Mem2Seq (Madotto et al., 2018) exhibits satisfactory performance when tested on the training KB. It represents the dialog history and the KB knowledge as a bag of words in a flat memory arrangement. This enables Mem2Seq to revisit each word several times, as needed, obtaining good performance. But at the same time, flat memory prevents it from capturing any surrounding context  this deteriorates its performance rapidly when the amount of new unseen information in the KB increases, as shown in Figure 1. On the other hand, the performance of copy augmented sequence-to-sequence network (Seq2Seq+Copy) (Eric and Manning, 2017), is robust to changes in the KB, but fails to achieve acceptable task-oriented performance. It captures context by representing the entire dialog history as one continuous sequence . However, it can be difficult for a sequence encoder to reason over long dialogs found in real-world datasets and its ability to learn the task gets hampered. We propose BOSSNET , a novel network that effectively disentangles the language and knowledge models, and also achieves state-of-the-art performance on three existing datasets. To achieve this, BOSSNET makes two design choices. First, it encodes the conversational input as a bag of sequences (BOSS ) memory, in which the input representation is built at two levels of abstraction. The higher level flat memory encodes the KB tuples and utterances to facilitate effective inferencing over them. The lower level encoding of each individual utterance and tuple is constructed via a sequence encoder (Bi-GRU). This enables the model to maintain the sequential context surrounding each token, aiding in better interpretation of unseen tokens at test time. Second, we augment the standard cross-entropy loss used in dialog systems with an additional loss term to encourage the model to only copy KB tokens in a response, instead of generating them via the language model. This combination of sequence encoding and additional loss (along with dropout) helps in effective disentangling between language and knowledge. We perform evaluations over three datasets  bAbI (Bordes and Weston, 2017), CamRest (Wen et al., 2016), and Stanford Multi-Domain Dataset (Eric et al., 2017). Of these, the last two are real-world datasets. We find that BOSSNET is competitive or significantly better on standard metrics in all datasets as compared to state-of-the-art baselines. We also introduce a knowledge adaptability (KA) evaluation, in which we systematically increase the percentage of previously unseen entities in the KB. We find that BOSSNET is highly robust across all percentage levels. Finally, we also report a human-based evaluation and find that BOSSNET responses are frequently rated higher than other baselines. Overall, our contributions are: 1. We propose BOSSNET , a novel architecture to disentangle the language model from knowledge incorporation in task-oriented dialogs. 2. We introduce a knowledge adaptability evaluation to measure the ability of dialog systems to scale performance to unseen KB entities. 3. Our experiments show that BOSSNET is competitive or significantly better, measured via standard metrics, than the existing baselines on three datasets. what is the food type they serve ? they serve indian food . We release our code and knowledge adaptability (KA) test sets for further use by the research community. 1 2 The BOSSNET Architecture The proposed Bag-of-Sequences Memory Network has an encoder-decoder architecture that takes as input (1) dialog history, which includes a sequence of previous user utterances { c u 1 , . . . , c un } and system responses { c s 1 , . . . , c sn  1 } , and (2) KB tuples { kb 1 , . . . , kb N } . The network then generates the next system response c sn = (cid:104) y 1 y 2 . . . y T (cid:105) word-by-word. The simplified architecture of BOSSNET is shown in Figure 2. In this section, we first describe the BOSS memory which contains the dialog history and KB tuples, followed by how the memory is consumed by the encoder and the decoder. We finally define the loss function, which, along with dropout, enables disentangled learning of language and knowledge. The memory M contains the dialog history { c u 1 , c s 1 , . . . , c un  1 , c sn  1 } and the KB tuples { kb 1 , . . . , kb N } . Each utterance in the dialog history and each KB tuple is placed in a memory cell. As utterances and tuples are inherently a sequence, we represent each memory cell m i as 1 https://github.com/dair-iitd/BossNet an ordered sequence of tokens (cid:104) w 1 i w 2 i . . . w | m i | i (cid:105) . For an utterance, the word tokens are followed by a temporal indicator and a speaker indicator { $u, $s } . For example, { good, morning, #1, $s } indicates this was the first utterance by the system. For a KB tuple, the tokens are sequenced as { subject, predicate, object } followed by temporal indicator and a kb indicator ( $db ). Token representation is generated using a bidirectional GRU. Let the outputs of the forward and backward GRUs for the token w ji be denoted as  h ji and  h ji respectively. Then the token representation  ( w ji ) is given by Eq. 1. Memory cell representation  ( m i ) is computed by concatenating the forward GRU output of its last token and the backward GRU output of its first token as in Eq. 2.  ( w ji ) = [  h ji ;  h ji ] (1)  ( m i ) = [  h | m i | i ;  h 1 i ] (2) 2.2 The BOSSNET Encoder The encoder used in BOSSNET is similar to the multi-hop attention encoder with layer-wise weights proposed by Sukhbaatar et al. (2015). The encoder in Sukhbaatar et al. (2015) uses two different embedding matrices, whereas we use just one to reduce the number of parameters. The encoder considers the last user utterance as the query q =  ( c un ) and computes the reduced representation q r using the memory M as follows: p i = softmax ( q T  ( m i )) (3) o = W r (cid:88) i p i  ( m i ) (4) q r = o + W o q (5) where W r , W o  R d  d are learnable parameters. The hop step can be re-iterated, by assigning the output of the previous hop as the new input query, i.e., setting q = q r . The output of the encoder after K hops, q kr , is assigned as the initial state of the BOSSNET decoder. BOSSNET models a copy-augmented sequence decoder, which generates the response one word at a time. At any decode time step t , the decoder can either generate a word from the decode vocabulary or copy a word from the memory. Consequently, the decoder computes: (1) generate distribution P g ( y t ) over the decode vocabulary, and (2) copy distribution P c ( y t ) over words in the memory. The generate distribution is computed using a standard sequence decoder (Sutskever et al., 2014) by attending (Luong et al., 2015) over the memory cell representations  . The copy distribution is generated by using a two-level attention . Given the decoder state s t , it first computes attention  t over the memory cells. Then it computes attention over the tokens in each memory cell m i . Finally it multiplies both these attentions to compute P c ( y t ) as follows:  ti = softmax ( s t  ( m i )) (6) e tij = s t  ( w ji ) (7)  tij =  ti  exp( e tij ) (cid:80) k exp( e tik ) (8) P c ( y t = w ) = (cid:88) ij : w ji = w  tij (9) The copy and generate distributions are combined using a soft gate g ts  [0 , 1] as in See et al. (2017). g ts is a function of the decoder state at time t and the word decoded in the previous time step. The decoder is trained using cross-entropy loss. The loss per response is defined as: L ce =  T (cid:88) t =1 log (cid:16) g ts P g ( y t ) + (1  g ts ) P c ( y t ) (cid:17) (10) where T is the number of words in the sequence to be generated and y t is the word to be generated at time step t . The decision to generate or copy is learnt implicitly by the network. However, to attain perfect disentanglement, the KB words should be copied, while the language should be generated. In other words, any word in the response that is present in the BOSSKB memory should have a low g s . To obtain this behavior, we define a disentangle label D l for each word in the response. This label is set to 1 if the word is present in the BOSSKB memory and 0 otherwise. We define a disentangle loss as follows: L d =  T (cid:88) t =1 g ts log D tl +(1  g ts ) log (1  D tl ) (11) We randomly drop some words with disentangle label set to 1 . This Disentangle Label Dropout (DLD) works in tandem with the disentangle loss and BOSS memory  it encourages the model to copy KB words whenever possible, based on their surrounding words. The relative weight of L d in the overall loss is controlled using a hyper-parameter (  ). The dropout rate is also a hyper-parameter. We perform experiments on three task-oriented dialog datasets: bAbI Dialog (Bordes and Weston, 2017), CamRest (Wen et al., 2016), and Stanford Multi-Domain Dataset (Eric et al., 2017). bAbI Dialog consists of synthetically generated dialogs with the goal of restaurant reservation. The dataset consists of five different tasks, all grounded to a KB. This KB is split into two mutually exclusive halves. One half is used to generate the train, validation, and test sets, while the other half is used to create a second test set called the OOV test set. CamRest is a human-human dialog dataset, collected using the Wiz-of-Oz framework, also aimed at restaurant reservation. It is typically used to evaluate traditional slot filling systems. In order to make it suitable for end-to-end learning, we stripped the handcrafted state representations and annotations in each dialog, and divided the 676 available dialogs into train, validation, and test sets (406, 135, and 135 dialogs, respectively). Stanford Multi-Domain Dataset (SMD) is another human-human dialog dataset collected using the Wiz-of-Oz framework. Each conversation is between a driver and an in-car assistant. The other datasets consist of dialogs from just one domain (restaurant reservation), whereas SMD consists of dialogs from multiple domains (calendar scheduling, weather information retrieval, and navigation). Each bAbI dialog task has an additional OOV test set, which helps to evaluate a model's robustness to change in information in the KB. A model that perfectly disentangles language and knowledge should have no drop in accuracy on the OOV test set when compared to the non-OOV test set. To measure the degree of disentanglement in a model, we generated 10 additional test sets for each real-world corpus by varying the percentage (in multiples of 10) of unseen entities in the KB. We systematically picked random KB entities and replaced all their occurrences in the dialog with new entity names. We will refer to these generated dialogs as the Knowledge Adaptability (KA) test sets. We compare BOSSNET against several existing end-to-end task-oriented dialog systems. These include retrieval models, such as the query reduction network (QRN) (Seo et al., 2017), memory network (MN) (Bordes and Weston, 2017), and gated memory network (GMN) (Liu and Perez, 2017). We also compare against generative models such as a sequence-to-sequence model (Seq2Seq), a copy augmented Seq2Seq (Seq2Seq+Copy) (Gulcehre et al., 2016), and Mem2Seq (Madotto et al., 2018). 2 For fairness across models, we do not compare against key-value retrieval networks (Eric et al., 2017) as they simplify the dataset by canonicalizing all KB words in dialogs. We noticed that the reported results in the Mem2Seq paper are not directly comparable, as they pre-processed 3 training data in SMD and bAbI datasets. For fair comparisons, we re-run Mem2Seq on the original training datasets. For completeness we mention their reported results (with pre-processing) as Mem2Seq*. We evaluate BOSSNET and other models based on their ability to generate valid responses. The per-response accuracy (Bordes and Weston, 2017) is the percentage of generated responses that exactly match their respective gold response. The per-dialog accuracy is the percentage of dialogs with all correctly generated responses. These accuracy metrics are a good measure for evaluating datasets with boilerplate responses such as bAbI. To quantify performance on other datasets, we use BLEU (Papineni et al., 2002) and Entity F1 (Eric and Manning, 2017) scores. BLEU measures the overlap of n-grams between the generated response and its gold response and has become a popular measure to compare task-oriented dialog systems. Entity F1 is computed by micro-F1 over KB entities in the entire set of gold responses. 2 We thank the authors for releasing a working code at https://github.com/HLTCHKUST/Mem2Seq 3 Mem2Seq used the following pre-processing on the data: 1) The subject (restaurant name) and object (rating) positions of the rating KB tuples in bAbI dialogs are flipped 2) An extra fact was added to the navigation tasks in SMD which included all the properties (distance, address, etc.) combined together as the subject and poi as the object. See Appendix. We use two human evaluation experiments to compare (1) the usefulness of a generated response with respect to solving the given task, and (2) the grammatical correctness and fluency of the responses on a 03 scale. We obtain human annotations by creating Human Intelligence Tasks (HITs) on Amazon Mechanical Turk (AMT). For each test condition (percentage of unseen entities), we sampled 50 dialogs from Camrest and SMD each, and two AMT workers labeled each system response for both experiments, resulting in 200 labels per condition per dataset per system. We evaluate four systems in this study, leading to a total of 1600 labels per condition. The detailed setup is given in the Appendix. We train BOSSNET using an Adam optimizer (Kingma and Ba, 2014) and apply gradient clipping with a clip-value of 40. We identify hyperparameters based on the evaluation of the held-out validation sets. We sample word embedding, hidden layer, and cell sizes from { 64, 128, 256 } and learning rates from { 10  3 , 5  10  4 , 10  4 } . The hyper-parameter  in the loss function is chosen between [0-1.5]. The Disentangle Label Dropout rate is sampled from { 0.1, 0.2 } . The number of hops for multi-hop attention in the encoder is sampled from { 1, 3, 6 } . The best hyper-parameter setting for each dataset is reported in the Appendix. Our experiments evaluate three research questions. 1. Performance Study : How well is BOSSNET able to perform the tasks of our three datasets as compared to the baseline models? 2. Disentanglement Study : How robust are the models in generalizing on the KA test sets? 3. Ablation Study : What is the performance gain from each novel feature in BOSSNET ? Table 1 reports the per-response and per-dialog (in parentheses) accuracies on the bAbI dialog tasks. The multi-hop retrieval-based models such as QRN, MN and GMN perform well on the non-OOV test sets for tasks 1, 2, and 5, but fail to exhibit similar performance on the corresponding OOV test sets. This result is expected as these models are trained to retrieve from a pre-defined set of responses. Their poor non-OOV performance on tasks 3 and 4 is attributed to an error in the bAbI dataset construction, due to which, the non-OOV and OOV test conditions are the same for these tasks (see Appendix). A simple generative model (Seq2Seq) achieves accuracies comparable to the multi-hop retrieval models. Enabling it with the ability to copy from the context (Seq2Seq+Copy) shows a considerable increase in performance, especially on the OOV test sets (and non-OOV tests for tasks 3 and 4). The strong performance of simple sequence encoders when compared with multi-hop encoders (in retrieval models) raises a question about the value of multi-hop inference. Mem2Seq answers this question, by obtaining improvements in several tasks, specifically on their OOV test sets. This clearly shows that multi-hop inference and the copy mechanism are essentials for task-oriented dialogs. Despite gains from the Mem2Seq model, the performance difference between the non-OOV and OOV test sets remains large. BOSSNET succeeds to bridge this gap with its ability to better interpret unseen words, using their surrounding context. It obtains significant improvements on average of about 34% per-dialog accuracy and 10% per-response accuracy for the bAbI OOV test sets. In Table 2, we report results on the real-world datasets. BOSSNET greatly outperforms other models in both Entity F1 metric and BLEU scores on CamRest. On SMD, BOSSNET achieves the best only in Entity F1. On further analysis of the generated responses we observe that BOSSNET responses often convey the necessary entity information from the KB. However, they consist of meaningful phrases with little lexical overlap with the gold response, reducing the BLEU scores. We investigate this further in our human evaluation. Human Evaluation: We summarize the human evaluation results for real-world datasets in Table 3. BOSSNET shows the best performance on Camrest, and is judged useful 77 times out of 100. Also, it has the highest average grammatical correctness score of 2.28 (very close to Seq2Seq and Mem2Seq). BOSSNET performs on par with Mem2Seq and Seq2Seq in its ability to relay appropriate information to solve SMD dialog tasks, and has a slightly higher grammaticality score. Figures 3 and 4 show the per-response accuracies of the two bAbI dialog tasks plotted against the percentage of unseen entities in KA sets. From Figure 3 we observe that BOSSNET remains immune to any variablity in the KB content, whereas the performance of Mem2Seq and Seq2Seq models drops drastically due to their inability to capture semantic representations of the injected KB entities. We see a similar trend in Figure 4, but here all the models show a drop in performance, with BOSSNET appearing the most steady. We explain this trend using the example dialog in Table 4. In the current dialog context, the system is required to provide the address of the selected restaurant, but since more than one restaurant in the KB is unseen, it becomes ambiguous for the network to identify KB (restaurantaddress) r bangkok overpriced thai 8  r bangkok overpriced thai 8 addr r bangkok overpriced thai 7  r bangkok overpriced thai 7 addr r bangkok overpriced thai 4  r bangkok overpriced thai 4 addr r bangkok overpriced thai 2  r bangkok overpriced thai 2 addr usr-1 may i have a table in an overpriced price range for nine people with thai food in bangkok ? the correct restaurant and infer its address. In the end, the system is forced to pick a random address  the probability of which being correct decreases as more restaurants become unseen. The performance on the CamRest KA test sets is illustrated in Figures 1 and 5. BOSSNET has the best performance with even a slight increase in both BLEU and Entity F1 metrics as more OOV content is injected in the dialog, probably because it is clear that it needs to copy when processing unseen entities. Seq2Seq+Copy is unable to perform well in CamRest as the length of the input (dialog history + KB tuples) is long and the size of the training set is also small. We believe that Seq2Seq+Copy works best in an environment with an abundance of short dialog training data (e.g., bAbI task 1 in Figure 3). SMD consists of dialogs with a large KB and a highly varying response pattern. This makes it very difficult to learn the language model  reflected in the low BLEU scores for all the systems. BOSSNET still provides the best F1 entity score due to CamRest SMD Info Grammar Info Grammar Seq2Seq 26 2.28 22 2.44 Seq2Seq+Copy 22 1.22 16 1.04 Mem2Seq 35 2.06 26 1.9 BOSSNET 80 2.44 51 2.28 Table 5: AMT Evaluations on CamRest and SMD (50% unseen) KA datasets its ability to inference efficiently on the large KB (Figure 6). Mem2Seq shows the best BLEU score performance on the original test set, but its performance drop of 42.5%, from 10.3 at 0% unseen to 5.93 at 100% unseen, is a lot heavier than that of BOSSNET which only drops 7.6%  8.27 at 0% unseen to 7.64 at 100% unseen. Human Evaluation: We summarize the human evaluation results for real-world datasets on the 50% unseen KA test set in Table 5. BOSSNET again outperforms the baselines and is labeled successful twice more often than the next best model on both Camrest and SMD. Seq2Seq appears to produce better sentence structures on the SMD dataset, primarily because it does not attempt to learn inference on the KB, allowing it to solely focus on learning the language model better. We assess the value of each model element, by removing it from BOSSNET . Table 6 reports the per-response accuracy scores for various configurations of BOSSNET on bAbI dialog tasks. It also reports the BLEU and entity F1 metric of various configurations on CamRest. Without BoSs Memory: This configuration uses the Bag-of-Bags (BoB) Memory rather than BOSS memory. The BoB memory is a simplified representation, similar to the one in the original Memory Networks. Here the token representation is the vector embedding of the token with no influence from the surrounding words and the memory cell representation is the sum of all its token embeddings. As a result, each word w representation is influenced equally by all words in a memory cell, irrespective of its distance from w . This makes capturing context in the immediate neighbourhood harder. Inability to capture the correct context prevents the configuartion from generalizing to OOV test sets. Without Disentangled Loss: Disentangled Loss ( L d ) plays an important role in enforcing that KB words be copied and other language be generated. By removing this loss component, it achieves better BLEU score in CamRest, but with a drop in Entity F1. Without the disentangled loss, the model sometimes learns to generate KB words. This severely affects OOV performance. As described earlier, an error in bAbI dataset construction tasks 3 and 4 effectively injects the validation set with a lot of OOVs. This anomaly in conjunction with the dropout (DLD), helps the configuration in achieving an acceptable performance for those tasks. Without Disentangled Label Dropout: BOSSNET learns to generate language and copy KB words. Without DLD, the model learns to memorize words to be copied rather than learning the context under which a word should be copied. Hence, the performance on OOV test sets is much inferior compared to the non-OOV setting. Overall, we notice that combining all three model elements is necessary in obtaining the best performance across all tasks. Table 7, demonstrates the ability of BOSSNET to copy entities (restaurant name and address) in its response. The other baselines either generate unwanted or irrelevant entities in their response, or fail to copy altogether. BOSSNET also best captures the language model effectively with a slight paraphrasing of the gold response. Table 8 contains only unseen entities. This example highlights the shortcomings of the Seq2Seq model as it ends up predicting a restaurant encountered during training. Mem2Seq copies a restaurant name without learning to sort the restaurants based on rating. BOSSNET , with its efficient memory addressing, is seen to be able to solve both issues. Compared to the traditional slot-filling based dialog (Williams and Young, 2007; Wen et al., 2017; Williams et al., 2017), end-to-end training methods (e.g., (Bordes and Weston, 2017), this work) do not require handcrafted state representations and their corresponding annotations in each dialog. Thus, they can easily be adapted to a new domain. We discuss end-to-end approaches along two verticals: 1) decoder: whether the response is retrieved or generated and 2) encoder: how the dialog history and KB tuples are encoded. Most of the existing end-to-end approaches retrieve a response from a pre-defined set (Bordes and Weston, 2017; Liu and Perez, 2017; Seo et al., 2017). These methods are generally successful when they have to provide boilerplate responses  they cannot construct responses by using words in KB not seen during training. Alternatively, generative approaches are used where the response is generated one word at a time (Eric and Manning, 2017; Madotto et al., 2018). These approaches mitigate the unseen entity problem by incorporating the ability to copy words from the input (Vinyals et al., 2015; Gu et al., 2016). The copy mechanism has also found success in summarization (Nallapati et al., 2016; See et al., 2017) and machine translation (Gulcehre et al., 2016). BOSSNET is also a copy incorporated generative approach. dialog history as a sequence (Eric and Manning, 2017; Gulcehre et al., 2016). Unfortunately, using a single long sequence for encoding also enforces an order over the set of KB tuples making it harder to perform inferencing over them. Other approaches represent the dialog context as a bag. Original Memory Networks (Bordes and Weston, 2017) and its extensions encode each memory element (ut-terance) as an average of all constituent words  this cannot point to individual words, and hence cannot be used with a copy mechanism. Mem2Seq encodes each word individually in a flat memory. Unfortunately, this loses the contextual information around a word, which is needed to decipher an unseen word. In contrast, BOSSNET uses a bag of sequences encoding, where KB tuples are a set for easier inference, and also each utterance is a sequence for effectively learning when to copy. KB (restaurantcuisineaddressphone) pizza hut fen ditton  italian  cambridge retail park newmarket road fen ditton  01223 323737 usr-1 may i have information for an italian restaurant in the east part of town ? We propose BOSSNET for training task-oriented dialog systems in an end-to-end fashion. BOSSNET combines a novel bag of sequences memory for storing a dialog history and KB tuples, with a copy-augmented generative decoder to construct dialog responses. It augments standard cross entropy loss of a sequence decoder with an additional term to encourage the model to copy KB words. BOSS memory and new loss term, in conjunction with a disentangle label dropout, enables the decoder to disentangle its language and knowledge models. BOSSNET achieves the state of the art results on bAbI dialog dataset, outperforming existing models by 10 points or more in its OOV conditions. In the knowledge adaptability test, we find that BOSSNET is highly robust to increasing the percentage of unseen entities at test time, suggesting a good language-knowledge disentanglement. Human evaluations show that BOSSNET responses are highly informative and slightly more grammatical compared to baselines. We will release our code and all curated datasets for further research. We thank Danish Contractor, Gaurav Pandey and Sachindra Joshi for their comments on an earlier version of this work. This work is supported by IBM AI Horizons Network grant, an IBM SUR award, grants by Google, Bloomberg and 1MG, and a Visvesvaraya faculty award by Govt. of India. We thank Microsoft Azure sponsorships, and the IIT Delhi HPC facility for computational resources.
abstain abstain abstain objective abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain objective abstain method abstain result abstain result result objective objective method result abstain abstain abstain other abstain abstain objective method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method method abstain method method method method method method abstain abstain abstain method abstain abstain other abstain method method method method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain result result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain method other method other other other other other other other other other other other other other other objective abstain abstain abstain abstain result abstain abstain other other other other