Despite recent improvements in open-domain dialogue models, state-of-the-art models are trained and evaluated on short conversations with little context. In contrast, the long-term conversation setting has hardly been studied. In this work we collect and release a human-human dataset consisting of multiple chat sessions whereby the speaking partners learn about each other's interests and discuss the things they have learnt from past sessions. We show how existing models trained on existing datasets perform poorly in this long-term conversation setting in both automatic and human evaluations, and we study long-context models that can perform much better. In particular, we find retrieval-augmented methods and methods with an ability to summarize and recall previous conversations outperform the standard encoder-decoder architectures currently considered state-of-the-art. Improvements in the ability to train large neural language models, together with the availability of larger and higher quality dialogue datasets, are spurring the development of increasingly convincing open-domain dialogue models (McTear, 2020). Unfortunately, a major aspect missing from the current state of the art is that human conversations can take place over long time frames, whereas the currently used systems suffer in this setting. Commonly used training and evaluation resources  while large in terms of number of training examples  include only short conversations, typically between 2-15 turns, consisting of a single conversational session. Perhaps for that reason, the current state-of-the-art models such as Meena (Adiwar-dana et al., 2020) and BlenderBot (Roller et al., 2020) employ Transformers with token truncation lengths of only the 128 most recent tokens, and  We use this term colloquially, see Agranoff et al. (1965) for evidence of goldfish long-term memory. are clearly incapable of incorporating long-term conversational context. Consequently, it is unclear how well these models will perform on long or multi-session open-domain conversations. In contrast, a successfully deployed bot will engage in many conversations over a length of time, as capturing organic user interest will garner continual reengagement from returning users. Long-term open-domain communication gives the opportunity for the conversation to develop and even improve with time as the model has more context and more understanding of that specific user's interests. However current models, due to context truncation, will never use this information. In this work we study methods for long-term open-domain conversation. As to the best of our knowledge no public domain task exists to study such methods, we collect and release 1 a new English dataset, entitled Multi-Session Chat (MSC) that consists of human-human crowdworker chats over 5 sessions, with each session consisting of up to 14 utterances, where the conversationalists reengage after a number of hours or days and continue chatting. Previous sessions are annotated with summaries of important personal points that may be useful in further conversations. When reengaging, conversationalists often address existing knowledge about their partner to continue the conversation in a way that focuses and deepens the discussions on their known shared interests, or explores new ones given what they already know. We study the performance of two long-context conversational architectures on this task: (i) retrieval-augmented generative models (Lewis et al., 2020b; Shuster et al., 2021); and (ii) a proposed read-write memory-based model that summarizes and stores conversation on the fly. We show that both techniques outperform conventional encoder-decoder Transformers, and that training 1 Dataset, model weights and code for this entire project will be made available upon acceptance. models on our new task give long-term conversational abilities that existing state-of-the-art models lack, as shown in both automatic metrics and human evaluations. We provide extensive experiments and ablations that study the reasons behind these improvements. A relatively large and growing number of either natural or crowdsourced datasets have been collected and used in open-domain dialogue research. These datasets focus on the vast array of different skills required by a dialogue agent, but conversations lengths are typically short. Recent state-of-the-art open-domain dialogue agents have utilized Daily Dialogue (Li et al., 2017), PersonaChat (Zhang et al., 2018), Empathetic Dialogues (Rashkin et al., 2019), Wizard of Wikipedia (Dinan et al., 2019) and Pushshift.io Reddit (Baumgartner et al., 2020); see Huang et al. (2020) for a review of other datasets. The number of conversational turns in these datasets is in the range of 2-15 turns, we provide statistics of some of these datasets in Table 2. We note there also exist some other kinds of dialogue datasets, e.g. from fantasy role-playing (Urbanek et al., 2019; Rameshkumar and Bailey, 2020) and TV shows as well (Poria et al., 2018). Crowdsourcing long conversations is difficult due to both the expense and the difficulty of employing crowdworkers for long lengths of time due to so called Human Intelligence Tasks (HITs) being typically of a short duration  only a few minutes (Paolacci et al., 2010). While organic long conversations regularly transpire on the internet, e.g. on messaging platforms, these are proprietary, and privacy concerns make public release implausible. Several existing datasets explore the use of personal knowledge used as context to dialogue, which can be seen as a short, simple memory provided to the bot. In Mazar et al. (2018) such personas were extracted from Reddit and used to train agents. In Zhang et al. (2018) personas were first crowdsourced, and speakers were asked to play those roles. Other works have considered encoding personas into vector-based weights (Li et al., 2016). In this work, we explore summarizing the long-term conversations that occur in order to store useful information about them. Summarization is a rich field where the vast majority of work focuses on summarizing documents (Kaikhah, 2004; Krys-cinski et al., 2019; Cheng and Lapata, 2016), for example summarizing in order to predict other relevant information (West et al., 2019), while there is some work on dialogue as well (Goo and Chen, 2018; Gliwa et al., 2019; Pan et al., 2018). Standard Transformers have a fixed context length which due to the all-vs-all self-attention mechanism becomes inefficient when it is too large. Consequently, many existing pre-trained models have short token truncation lengths, e.g. 128 tokens, as in BlenderBot (Roller et al., 2020) and Meena (Adiwardana et al., 2020), or 1024 tokens, as in BART (Lewis et al., 2020a). A number of approaches have been proposed to ameliorate this issue. Long-context Transformers consider ways to speed up the self-attention mechanism (Child et al., 2019; Kitaev et al., 2019; Beltagy et al., 2020) and retrieval-augmented methods consider ways to select the pertinent parts of the context to consider (Dinan et al., 2019; Lewis et al., 2020b; Shuster et al., 2021) which can also be related to earlier neural QA methods (Chen et al., 2017). To conduct research on long-term conversations, we require data to both train on and to evaluate models. We consider the natural case where two speakers chat online in a series of sessions as is for example common on messaging platforms. Each chat session consists of 6-7 turns for each speaker. Then, after a certain amount of (simulated) time has transpired, typically hours or days, the speakers resume chatting, either continuing to talk about the previous subject, bringing up some other subject from their past shared history, or sparking up conversation on a new topic. We consider this multi-session long conversation setup, and name our dataset Multi-Session Chat (MSC). Data Collection To build our publicly available dataset we employ crowdworkers. We provide screenshots of the task, and details of quality control via onboarding, crowdworker co-rating, and automatic evaluation procedures in Appendix B. Personas Crowdworkers are asked to play a role, rather than speaking about their own personality, which helps mitigate privacy concerns, and ensures diversity even if the same crowdworker conducts multiple conversations. In addition to the crowdworkers being specifically told to play the role, they are also told not to discuss aspects of their real profiles or indeed any personally identifiable informa-5181 Train Valid Test Data Type Epsiodes Utts. tion. The role is provided as a series of sentences describing characteristics, events and opinions of the character they are playing. We use the 1,155 personas crowdsourced from Zhang et al. (2018), validation and test use separate personas from the ones used in the training set. Session 1 For the first chat session we use the PERSONACHAT dataset (Zhang et al., 2018), which already involves short conversations where two speakers get to know each other for the first time. We note that these conversations rarely go beyond the superficial stage because speakers simply do not have enough turns to discuss any topic deeply. Sessions 2, 3, 4, . . . For subsequent sessions, we first select a random amount of (simulated) time that has elapsed since the previous session, chosen to be either 1-7 hours or 1-7 days, as ideally speakers would reengage within that timeframe. We ask the crowdworkers to play the same roles that were played in the previous session, acting as if that amount of time has transpired. We note these crowdworkers may not be the same ones that played those characters in previous sessions, but will be playing the same roles: this makes the task tractable in a crowdworking frameworking where jobs are typically short, and matching pairs over a long duration would be infeasible. We instruct the workers to chitchat with another worker for 6 turns, as if you were catching up since last time you two spoke. and that When you expand the topic, make sure it makes sense with the personal details already mentioned., i.e. emphasizing that not only must they play their role, but also pay attention to previous interactions with the other speaker. Session Lengths We collect two lengths of training conversation: 4000 episodes with 3 sessions, and 1001 episodes with 4 sessions. For the validation and test data, the sessions extend up to 5 sessions, giving us a way to measure long-context session performance that extends beyond the training set distribution. Conversation Summaries (Extended Personas) We give crowdworkers access to all previous dialogues between the two conversational roles (for the role they are playing, and their partner's role). However, as the conversation gets longer, this becomes infeasible to read and digest within a limited amount of time. Therefore, between each session, including after session 1, we run a separate crowdworker task in which conversations are summarized into important points, which are much shorter than the full dialogues themselves. We then show previous dialogues, along with these summaries, as the primary reference for subsequent session dialogues. As these summaries were collected in order to store the important points pertinent to either one or the 5182 other speaker, they can also be seen to function as extensions of the original given personas. As the two speakers continue to converse they create more depth to those characters. Dataset Examples Two dataset examples, which consist of four sessions each, along with example summary annotations, are given in Appendix C (provided in the Appendix due to their length). Dataset Statistics Statistics of the multi-session chat dataset are given in Table 1 and a comparison with other standard open-domain dialogue datasets is given in Table 2. We can see that the number of training utterances per episode is larger than other datasets (last column of Table 2). Our multi-session training chats that last 4 sessions have an average of  53 utterances in a full conversation (over all sessions), while our validation and test chats over 5 sessions have an average of  66 utterances. In contrast, other standard datasets are in the range of 2.6-14.7 utterances on average. This brings challenges in open-domain dialogue modeling due to the large context size, e.g. an average of 1614 tokens as tokenized by the BlenderBot BPE dictionary (Roller et al., 2020), where the Transformer used in that work has a truncation length of 128. Further information on the dataset including analysis of its quality is given in Appendix B. 4 Modeling Multi-Session Chat 4.1 Transformer Encoder-Decoders The most straight-forward approach for modeling dialogue using our new task is simply to use a large language model as is standard in open-domain dialogue, i.e. an encoder-decoder Transformer as in the Meena (Adiwardana et al., 2020) and BlenderBot (Roller et al., 2020) systems. We consider using the BST 2.7B parameter model from BlenderBot as an initial pre-trained model, which we then fine-tune on the Multi-Session Chat task. Encoder Truncation As BST 2.7B has a truncation of 128 tokens in the encoder, we consider extending this to a larger input. To do this, we extend its available positional encodings from 128 to 512 or 1024 tokens as we fine-tune the whole network on the downstream task. We add new positional embeddings to be trained such that the existing ones (the first 128 most recent tokens) do not change from before. We then evaluate the impact of these choices in order to select the best model. A popular technique when dealing with a large collection of text, only some of which is relevant, is to use a retrieval-augmented Transformer. A retrieval system is used to search over a text collection, and select some of it to be included in the final encoding which is attended to by the Transformer decoder. RAG The RAG (Retrieval-Augmented Generation) approach (Lewis et al., 2020b) utilizes a neural-retriever-in-the-loop which is itself a second Transformer. Documents to be retrieved are stored in an approximate nearest-neighbor FAISS index (Johnson et al., 2019), and a DPR (Dense Passage Retrieval) (Karpukhin et al., 2020) Transformer bi-encoder model is used to score document-context pairs in order to rank them based on their match, where the base DPR model is pre-trained on QA data pairs. The DPR model is thus used to both retrieve from the FAISS index, and then score the top N candidates. The entire system is trained end-to-end so that retrieval is optimized to help improve generation. This setup was shown to work for dialogue in particular in Shuster et al. (2021). FiD and FiD-RAG We also consider the Fusion-in-Decoder (FiD) (Izacard and Grave, 2020), another method that has been shown to perform well. In this approach, the pre-trained retriever is used directly: each of the top N documents returned is prepended to the context and encoded separately by the encoder, and finally all the results are concatenated. The decoder then attends to these encodings to produce a final response. We consider the pre-trained retriever to either be standard pre-trained DPR, or the RAG-trained retriever, called FiD-RAG (Shuster et al., 2021). Retriever and Documents In this work the set of passages in the memory is not large enough to require a FAISS index, but it is large enough that retrieval may be useful. We thus store for every item in the memory the vector encoding by the DPR model (whereas in the FAISS approach this dense vector is approximated instead). Then given a dialogue context, we score each memory using the bi-encoder, and use the top N for generation. In our case, the memories consist of dialog utterances from the history of the conversation. We consider the chunk (document) size as a hyperpa-rameter and try either encoding utterances as separate documents, or else whole sessions (or session summaries) as documents. The latter (whole se-5183 Pre-Train Model Truncation Sessions 1-4 Session 1 Session 2 Session 3 Session 4 Trunc% (S4) With no previous session context BST 2.7B 128 9.23 8.76 9.45 9.31 9.40 51% BST 2.7B 512 9.06 8.18 9.42 9.26 9.36 0% BST 2.7B 1024 9.08 8.20 9.46 9.29 9.37 0% With previous session dialogue context BST 2.7B 128 9.16 8.75 9.32 9.22 9.32 100% BST 2.7B 512 8.87 8.15 9.14 9.04 9.17 100% BST 2.7B 1024 8.89 8.17 9.18 9.05 9.16 80% With previous session summary context BST 2.7B 128 9.09 8.77 9.24 9.12 9.24 100% BST 2.7B 512 8.79 8.17 8.69 9.15 9.22 36% BST 2.7B 1024 8.80 8.18 9.05 8.91 9.04 0% Table 3: Comparison of different context truncation lengths and context types when training on MULTISESSIONCHAT . sions) worked better, and we report those in the final results. For N we try values 3, 5 and 6, and also choose the best for each method according to the validation set. The retrieval-augmentation models described in the previous section retrieve from the set of past dialogues. Simply storing historical text in the memory in their raw form is a simple approach that is often used elsewhere in the literature, e.g. in question answering or knowledge-grounded dialogue. However, those approaches have two potential drawbacks: (i) there is a lot of context to store, and hence retrieve from; (ii) no processing has been done on that content, so the reading, retrieving and combining operations required to generate an answer leave a lot of work for the model to do. We therefore propose instead a novel memory augmentation that first summarizes pertinent knowledge and only stores that in an attempt to solve both problems. 1. An encoder-decoder abstractive summarizer that takes as input the dialogue history, and outputs a summary of new pertinent information contained in the last dialogue turn, or no-summary if there is no new information found. When found, the summarized knowledge is added to the long-term memory. For (1) we can use the human annotated data from our newly collected MSC task to know what summaries to generate (see section 3 and Figure 1 in the Appendix). We thus train a supervised encoder-decoder model to produce summaries. For (2) we can use the same systems as presented in subsection 4.2 to both retrieve from the summarization memories, and to finally generate an appropriate response. That is, we store the summaries in documents and retrieve them using either RAG, FiD or FiD-RAG. Using session dialogue context We compare different context types in Table 3, evaluating over sessions 1-4. We observe an improvement in perplexity when incorporating the dialogue history from previous chat sessions, compared to no session context, for all sessions after the first one, and for all context lengths  with larger context lengths giving better improvement. This shows that our human conversationalists do use previous sessions to make dialogue more salient in successive sessions as this is reflected in the collected human-human dataset  and that our models are able to utilize this information well when training on this data. performance of using gold session summary contexts, as annotated by crowdworkers, in Table 3. As the summaries include salient points, they are potentially more informative than dialogue context for a generative model. We find perplexities improve when using summaries compared to using dialogue context (or no context at all) over all sessions after the first one, and for all context lengths, although the improvements are not large. This shows that conversation summaries are potentially useful for dialogue generation in the long-context case. Comparing performance on session openings Session openings in the MSC dataset look quite different to other dialogue datasets that do not have a session format. This is because they involve an opening message that is intended to reengage the other speaker after a period of time, using known information that has been exchanged between speakers. In Table 4 we compare models that use different context types on only these opening responses. In this case we find much more pronounced perplexity differences between no session context history, dialogue history or summary context history. For example, we see around around 2 perplexity points difference between using or not using previous session context. We show examples of opening session generations in Appendix C. We observe that opening messages are categorically different to other conversation turns, typically involving a statement or question given knowledge of shared interests contained in the long-context. This explains why collection of our new dataset is so important for this goal, as reflected in perplexity improvements. That is, they indicate that our new task will likely help improve multi-session conversational engagement with users compared to existing training schemes. Comparing different context lengths As shown in Table 3 changing the context length of a Transformer can impact the performance in our task. With no previous session context, improvements are minimal for sessions 2 onwards. However, using session dialogue or summary contexts we do see improvements with larger lengths of 512 or 1024 tokens, compared to 128. The last column of Table 3 shows the percentage of responses where the input to the Transformer is truncated for session 4, for each truncation length. One can see that using summaries can be beneficial as they are shorter, 5185 Model Session 1 Session 2 Session 3 Session 4 Session 5 Session Openings BST 2.7B (Roller et al., 2020) 8.97 9.98 10.26 10.40 10.50 12.92 MSC 2.7B (truncate 128) 8.87 8.89 9.10 9.21 9.27 8.95 MSC 2.7B (truncate 1024) 8.25 8.76 8.93 9.07 9.16 8.09 MSC 2.7B (RAG) 8.22 8.78 8.97 9.11 9.17 8.10 MSC 2.7B (FiD) 8.22 8.75 8.92 9.05 9.11 8.06 MSC 2.7B (FiD-RAG) 8.23 8.75 8.93 9.04 9.11 8.03 SumMem-MSC 2.7B (truncate 1024) 8.25 8.71 8.89 9.01 9.09 8.04 SumMem-MSC 2.7B (RAG) 8.24 8.81 9.00 9.10 9.17 8.05 SumMem-MSC 2.7B (FiD) 8.20 8.71 8.89 9.00 9.07 7.91 SumMem-MSC 2.7B (FiD-RAG) 8.22 8.70 8.89 9.00 9.07 7.87 Table 7: Test perplexity across sessions for our retrievaland memory-augmented models (bottom two blocks) compared to several encoder-decoder baselines (top three rows). Summary context performance We can ablate the summary model training data to understand its impact further, results of which are given in Table 4. We see that removing the time feature (indicating how long ago the previous session occurred) only has minimal effect. Removing either the partner or self summary (and keeping the other one), on the other hand, has a larger effect in both cases, where keeping the self summary is slightly more important. Keeping both features is best. These differences, as before, are magnified when looking at session opening performance. Predicted summary models We train models to predict dialogue summaries, and use predicted summaries of previous sessions as context (instead of the full dialogue history or the gold summary). The training data for predicting summaries consists of, for each turn, either a summarizing sentence or the no_summary label. As 42% of turns have the no_summary label, this can be overexpressed in the model at beam decoding time 2 , we therefore experiment with sampling this label only K % of the time during training in Table 5. Example predictions (for the 5% sampling model) are shown in Figure 1. We find that subsampling gives better results and closer sparsity levels to the original human annotated data (e.g., with K = 25% ). We compare predicted summaries with K = 5% sampling to other methods of modeling long-context in Table 4. We observe results that are between using a standard dialogue history (predicted summaries are slightly better), and using gold summaries (pre-dicted summaries are not as good). 2 We use a beam size of 3 and minimum beam length 10 with no context blocking. Varying the number of training sessions We vary the amount of available training sessions from 1-4, with results reported in Table 6. We observe large gains when using more than one training session compared to only one (around 1.5 perplexity points), again justifying the construction of our MSC training data. The gains however decrease with the number of available sessions, e.g. between having 1-3 training sessions vs. 1-4 only gives a 0.03 perplexity gain averaged across sessions. The gain even on session 4 is not that large despite the 1-4 training data being in-distribution, whereas 1-3 is not, in addition to 1-4 having more training data. Retrieval-augmentation model Comparison of our retrieval-augmented methods are given in Table 7, training on MSC using the BST 2.7B model as pre-training, hence called MSC 2.7B (RAG), (FiD) or (FiD-RAG), depending on the augmentation method. These methods are compared to the existing BlenderBot model (BST 2.7B), or training with MSC with no augmentation (MSC 2.7B with different dialogue history context truncation lengths). We find that all three retrieval augmentation methods, when using the session level-document size as retrieval documents, can effectively use retrieval to extend the conversation history length. We see a large performance improvement over the existing BlenderBot model or a truncation of 128 of the MSC 2.7B model. Performance improvements over MSC 2.7B with a truncation length of 1024 are minimal, but the retrieval-augmented models are guaranteed to have a memory that essentially never forgets the conversation, no matter how long it gets, whereas the truncation model does not. previous dialogue history is summarized before being stored in the model's long-term memory, called SumMem-MSC 2.7B. We use the RAG, FiD, or RAG-FiD methods to retrieve from that memory, or we compare to a fixed memory of 1024 tokens that is truncated, resulting in four different methods that we compare. Results are given in Table 7. While improvements are small, we see the same patterns as for the retrieval-augmented methods that SumMem-MSC 2.7B FiD-RAG is better than FiD which is in turn better than RAG, with FiD and FiD-RAG better than truncation at session openings. Moreover, all SumMem-MSC models outperform their retrieval-augmented model counterparts MSC 2.7B (RAG/FiD/FiD-RAG). SumMem-MSC 2.7B (FiD-RAG) thus provides the best results out of all methods tested in this work. Further Detailed Automatic Metrics Our analysis so far measured perplexity. We report more automatic metrics (F1 and BLEU) in Appendix A, which yield similar conclusions. We perform a human evaluation using crowdworkers. The conversations begin with two randomly chosen personas from the validation set, and one is assigned to the crowdworker who is asked to play that role. We select the conversation to be the 5 th session that these two speakers will converse, and make available the summary of the previous 4 sessions. We ask the crowdworkers to have a natural conversation, where they will also evaluate their partner's responses for conversational attributes, in particular whether they reference knowledge of their own or the other speaker's persona (or topics they discussed) from previous sessions, from the current session, or neither. On each turn of the conversation the crowdworker is asked to check all attribute boxes that apply. A screenshot can be found in Figure 6 in the Appendix showing the UI. Each conversation consists of 15 messages (7 from the human, 8 from the bot). At the end of the conversation, an additional question collects an overall engagingness score (out of 5) for their speaking partner. The results are given in Table 8. We find that MSC-trained models outperform BlenderBot (BST 2.7B) in terms of both per-turn engaging responses and final ratings. Further, our summarization memory models (all three variants RAG, FiD and FiD-RAG) outperform encoder-decoders with different levels of truncation of the dialogue history (MSC 2.7B with truncate 128 and 1024). For example, SumMem-MSC 2.7B (RAG) achieves an engaging response rate of 62.1% and final rating of 3.65, compared to BlenderBot's 53.0% and 3.14 and MSC 2.7B (truncate 1024)'s 54.2% and 3.47. For all MSC models, while rates of referencing their own topics are not particularly increased, we do observe increased rates of referencing partner topics from previous sessions, with higher rates for the summarization memory models. For example, 33.8% for SumMem-MSC 2.7B (RAG) compared to BlenderBot's 14.5%. This is likely an important reason why human raters feel the summarization memory models are more engaging. We have shown that existing dialogue models, both in terms of training data and models trained, fail to conduct long-term conversations adequately. Our work investigates recent model architectures to ameliorate this issue, and collects a new crowdsourced task, Multi-Session Chat to both train and evaluate these models. We show, in terms of both automatic metrics and human evaluations, that these long-context dialogue modeling approaches outperform the previous systems. Future work should investigate further improvements to architectures for the long-context dialogue setting. The dialogue models we use in this work utilize large language models, and therefore have similar concerns as in other work, in particular concerns about toxic language, bias and other issues during language generation (Bender et al., 2021). For open-domain dialogue in particular, see Xu et al. (2020); Dinan et al. (2021) for reviews of the literature and evaluation of recent methods that try to mitigate these safety issues. Our work focuses on models with long-term memory and open-domain conversations wherein speakers may divulge personal interests. We remark that, during data collection, crowdworkers were specifically playing roles with given personality traits, not talking about themselves, and hence not identifying any personal information. During conversations with our trained models, the models will store information they learn from the exchange. In contrast to current standard language models, our models have the capability of storing this in the long-term. This information is stored in the memory of the model, private to the individual's conversation, and hence is not shared with anyone else.
abstain abstain method result objective abstain abstain abstain result abstain abstain abstain abstain abstain method objective abstain abstain method abstain abstain result objective method other other other abstain other other other other other other objective other other other other other method method abstain abstain method method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain result abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain abstain result objective result abstain method abstain method method method abstain abstain