Text-based games provide an interactive way to study natural language processing. While deep reinforcement learning has shown effectiveness in developing the game playing agent, the low sample efficiency and the large action space remain to be the two major challenges that hinder the DRL from being applied in the real world. In this paper, we address the challenges by introducing world-perceiving modules, which automatically decompose tasks and prune actions by answering questions about the environment. We then propose a two-phase training framework to decouple language learning from reinforcement learning, which further improves the sample efficiency. The experimental results show that the proposed method significantly improves the performance and sample efficiency. Besides, it shows robustness against compound error and limited pre-training data. Text-based games are simulated environments where the player observes textual descriptions, and acts using text commands (Hausknecht et al., 2020; Urbanek et al., 2019). These games provide a safe and interactive way to study natural language understanding, commonsense reasoning, and dialogue systems. Besides language processing techniques, Reinforcement Learning has become a quintessential methodology for solving text-based games. Some RL-based game agents have been developed recently and proven to be effective in handling challenges such as language representation learning and partial observability (Narasimhan et al., 2015; Fang et al., 2017; Ammanabrolu and Riedl, 2019). Despite the effectiveness, there are two major challenges for RL-based agents, preventing them from being deployed in real world applications: the low sample efficiency , and the large action space (Dulac-Arnold et al., 2021). The low sample  Corresponding author efficiency is a crucial limitation of RL which refers to the fact that it typically requires a huge amount of data to train an agent to achieve human-level performance (Tsividis et al., 2017). This is because human beings are usually armed with prior knowledge so that they don't have to learn from scratch (Dubey et al., 2018). In a language-informed RL system, in contrast, the agent is required to conduct both language learning and decision making regimes, where the former can be considered as prior knowledge and is much slower than the later (Hill et al., 2021). The sample efficiency could be improved through pre-training methods, which decouple the language learning from decision making (Su et al., 2017). The selection of pre-training methods thus plays an important role: if the pre-trained modules perform poorly on unseen data during RL training, the incurred compound error will severely affect the decision making process. Another challenge is the large discrete action space: the agent may waste both time and training data if attempting irrelevant or inferior actions (Dulac-Arnold et al., 2015; Zahavy et al., 2018). In this paper, we aim to address these two challenges for reinforcement learning in solving text-based games. Since it is inefficient to train an agent to solve complicated tasks (games) from scratch, we consider decomposing a task into a sequence of subtasks as inspired by (Andreas et al., 2017). We design an RL agent that is capable of automatic task decomposition and subtask-conditioned action pruning, which brings two branches of benefits. First, the subtasks are easier to solve, as the involved temporal dependencies are usually short-term. Second, by acquiring the skills to solve subtasks, the agent will be able to learn to solve a new task more quickly by reusing the learnt skills (Bar-reto et al., 2020). The challenge of large action space can also be alleviated, if we can filter out the actions that are irrelevant to the current subtask. ings can understand the environment conditions through question answering (Das et al., 2020; Ammanabrolu et al., 2020), we design world-perceiving modules to realize the aforementioned functionalities (i.e., task decomposition and action pruning) and name our method as Q uestion-guided W orld-perceiving A gent (QWA) * . Fig. 1 (b) shows an example of our decision making process. Being guided by some questions, the agent first decomposes the task to obtain a set of available subtasks, and selects one from them. Next, conditioned on the selected subtask, the agent conducts action pruning to obtain a refined set of actions. In order to decouple language learning from decision making, which further improves the sample efficiency, we propose to acquire the world-perceiving modules through supervised pre-training. We design a two-phase framework to train our agent. In the first phase, a dataset is built for the training of the world-perceiving modules. In the second phase, we deploy the agent in games with the pre-trained modules frozen, and train the agent through reinforcement learning. We conduct experiments on a series of cooking games. We divide the games as simple games and complex games, and construct the pre-training dataset from simple games only. The experimental results show that QWA achieves high sample efficiency in solving complex games. We also show that our method enjoys robustness against compound error and limited pre-training data. Our contributions are summarized as follows: Firstly, we develop an RL agent featured with question-guided task decomposition and action space reduction. Secondly, we design a two-phase * Code is available at: https://github.com/ YunqiuXu/QWA framework to efficiently train the agent with limited data. Thirdly, we empirically validate our method's effectiveness and robustness in complex games. The RL agents for text-based games can be divided as text-based agents and KG-based agents based on the form of observations. Compared with the text-based agents (Narasimhan et al., 2015; Yuan et al., 2018; Adolphs and Hofmann, 2020; Jain et al., 2020; Yin and May, 2019; Xu et al., 2020a; Guo et al., 2020), which take the raw textual observations as input to build state representations, the KG-based agents construct the knowledge graph and leverage it as the additional input (Ammanabrolu and Riedl, 2019; Xu et al., 2020b). By providing structural and historical information, the knowledge graph helps the agent to handle partial observability, reduce action space, and improve generalizability across games. Based on how actions are selected, the RL agents can also be divided as parser-based agents, choice-based agents, and template-based agents. The parser-based agents generate actions word by word, leading to a huge combinatorial action space (Kohita et al., 2021). The choice-based agents circumvent this challenge by assuming the access to a set of admissible actions at each game state (He et al., 2016). The template-based agents achieve a trade-off between the huge action space and the assumption of admissible action set by introducing the template-based action space, where the agent selects first a template, and then a verb-object pair either individually (Hausknecht et al., 2020) or conditioned on the selected template (Ammanabrolu and Hausknecht, 2020). In this work, we aim to improve the sam-539 ple efficiency and reduce the action space through pre-training. Being agnostic about the form of observations and the action selecting methods, our work complements the existing RL agents. Our work is closely related to task decomposition (Oh et al., 2017; Shiarlis et al., 2018; Sohn et al., 2018) and hierarchical reinforcement learning (Dayan and Hinton, 1992; Kulkarni et al., 2016; Vezhnevets et al., 2017). Similar to our efforts, Jiang et al. (2019) and Xu et al. (2021) designed a meta-policy for task decomposition and subtask selection, and a sub-policy for goal-conditioned decision making. Typically, these works either assume the access to a set of available subtasks, or decompose a task through pre-defined rules, while we aim to achieve automatic task decomposition through pre-training, and remove the requirement for expert knowledge during reinforcement learning. Besides, existing work assumes that unlimited interaction data can be obtained to train the whole model through RL. In contrast, we consider the more practical situation where the interaction data is limited, and focus on improving the RL agent's data efficiency. Regarding the sub-policy, we do not assume the access to the termination states of the subtasks. We also do not require additional handcrafted operations in reward shaping (Bah-danau et al., 2019). There have been a wide range of work studying pre-training methods or incorporating pre-trained modules to facilitate reinforcement learning (Ey-senbach et al., 2018; Hansen et al., 2019; Sharma et al., 2019; Gehring et al., 2021; Liu et al., 2021; Schwarzer et al., 2021). One major branch among them is Imitation Learning (IL), where the agent is trained to imitate human demonstrations before being deployed in RL (Hester et al., 2018; Zhu et al., 2018; Reddy et al., 2019). Although we also collect human labeled data for pre-training, we leverage the data to help the agent to perceive the environment instead of learning the solving strategies. Therefore, we do not require the demonstrations to be perfect to solve the game. Besides, our method prevails when pre-trained on simple tasks rather than complicated ones, making it more feasible for human to interact and annotate (Arumugam et al., 2017; Mirchandani et al., 2021). Further discussions to compare our method with IL are provided in subsequent sections. In the domain of text-based games, some prior works have involved pre-training tasks such as state representation learning (Ammanabrolu et al., 2021; Singh et al., 2021), knowledge graph constructing (Murugesan et al., 2021) and action pruning (Hausknecht et al., 2019; Tao et al., 2018; Yao et al., 2020). For example, Ammanabrolu et al. (2020) designed a module to extract triplets from the textual observation by answering questions, and use these triplets to update the knowledge graph. As far as we know, we are the first to incorporate pre-training based task decompositon in this domain. Besides, instead of directly pruning the actions based on the observation, we introduce subtask-conditioned action pruning to further reduce the action space. POMDP Text-based games can be formulated as a Partially Observable Markov Decision Processes (POMDPs) (Ct et al., 2018). A POMDP can be described by a tuple G = S , A , P, r,  , O,   , with S representing the state set, A the action set, P ( s  | s, a ) : S  A  S (cid:55) R + the state transition probabilities, r ( s, a ) : S  A (cid:55) R the reward function,  the observation set, O the conditional observation probabilities, and   (0 , 1] the discount factor. At each time step, the agent receives an observation o t   based on the probability O ( o t | s t , a t  1 ) , and select an action a t  A . The environment will transit into a new state based on the probability T ( s t +1 | s t , a t ) , and return a scalar reward r t +1 . The goal of the agent is to select the action to maximize the expected cumulative discounted rewards: R t = E [ (cid:80)  t =0  k r t ] . Observation form In text-based games, the observation can be in the form of text, knowledge graph, or hybrid. Fig. 1 (a) shows an example of the textual observation and the corresponding KG-based observation. We do not make assumptions about the observation form and our method is compatible with any of those forms. Problem setting We aim to design an RL-based agent that is able to conduct automatic task decomposition and action pruning in solving text-based games. We consider games sharing similar themes and tasks, but varying in their complexities (Ad-hikari et al., 2020; Chen et al., 2021). Taking the cooking games (Ct et al., 2018) as an example, 540 Figure 2: Subtasks for solving the task is always make the meal. To accomplish this task, the agent has to explore different rooms to collect all ingredients, prepare them in right ways, and make the meal. A game's complexity depends on the number of rooms, ingredients, and the required preparation steps. We define a subtask as a milestone towards completing the task (e.g., get apple if apple is included in the recipe), and a subtask requires a sequence of actions to accomplish (e.g., the agent has to explore the house to find the apple). A game is considered simple, if it consists of only a few subtasks, and complex if it consists of more subtasks. Fig. 2 gives examples of simple games and complex games. While being closer to real world applications, complex games are hard to solve by RL agents because: 1) it's expensive to collect sufficient human labeled data for pre-training; 2) it's unrealistic to train an RL agent from scratch. We therefore focus on agent's sample efficiency and performance on complex games. Our objective is to leverage the labeled data collected from simple games to speed up RL training in complex games, thus obtaining an agent capable of complex games. For more details and statistics of the simple / complex games used in our work, please refer to Sec. 5.1. Fig. 3 shows the overview of our QWA agent. We consider two world-perceiving modules: a task selector and an action validator. Given the observation o t and the task candidate set T , we use the task selector to first obtain a subset of currently available subtasks T t  T , then select a subtask T t  T t . Given T t and the action candidate set A , we use the action validator to get an action subset A t  A , which contains only those relevant to the subtask T t . Finally, given o t and T t , we use an action selector to score each action a  A t , and the action with the highest score will be selected as a t . The training of the world-perceiving modules can be regarded as the language learning regime, while the training of the action selector can be regarded as the decision making regime. We consider a two-phase training strategy to decouple these two regimes to further improve the sample efficiency (Hill et al., 2021). In the pre-training phase, we collect human interaction data from the simple games, and design QA datasets to train the world-perceiving modules through supervised learning. In the reinforcement learning phase, we freeze the pre-trained modules, and train the action selector in the complex games through reinforcement learning. Depending on the experiment settings, T and A can be either fixed vocabulary sets (parser-based), or changing over time (choice-based). We regard a subtask available if it is essential for solving the global task, and there's no prerequisite subtask. For example, the subtask get apple in Fig. 1, as the object apple is an ingredient which has not been collected. Although another subtask dice apple is also essential for making the meal, it is not available since there exists a prerequisite subtask (i.e., you should collect the apple before dicing it). The aim of the task selector is to identify a subset of available subtasks T t  T , and then select one subtask T t  T t . We formulate the mapping f ( o t , T )  T t as a multi-label learning problem (Zhang and Zhou, 2013). For simplicity, we assume that the subtask candidates are independent with each other. Thus, the multi-label learning problem can be decomposed as |T | binary classification problems. Inspired by the recent progress of question-conditional probing (Das et al., 2020), language grounding (Hill et al., 2021), and QA-based graph construction (Ammanabrolu et al., 2020), we cast these binary classification problems as yes-or-no questions, making the task selector a world-perceiving module. For example, the corresponding question for the subtask candidate get apple could be Whether get apple' is an available sub-task?. This module can guide the agent to under-541 Figure 3: The overview of QWA. stand the environment conditions through answering questions, but will not directly lead the agent to a specific decision. We can obtain this module through supervised pre-training, and decouple it from reinforcement learning to yield better sample efficiency. Fig. 1 (b) shows some sample QAs, where a human answerer can be replaced by a pre-trained task selector. Some previous work also considered task decomposition (Chen et al., 2021; Hu et al., 2019), but the related module is obtained through imitating human demonstrations, which is directly related to decision making instead of world perceiving. Compared with these work, our method has two folds of benefits. First, there may exist multiple available subtasks at a timestep. Imitating human demonstrations will specify only one of them, which may be insufficient and lead to information loss. Second, we do not require expert demonstrations which guarantee to solve the game. Instead, we can ask humans to annotate either imperfect demonstrations, or even demonstrations from a random agent. We will treat the IL-based method as a baseline and conduct comparisons in the experiments. Given the set of available subtasks T t , arbitrary strategies can be used to select a subtask T t from it. For example, we can employ a non-learnable task scorer to obtain T t by random sampling, since each subtask T  T t is essential for accomplishing the task. We can also train a task scorer via a meta-policy for adaptive task selection (Xu et al., 2021). After obtaining the subtask T t , we conduct action pruning conditioned on it (or on both T t and o t ) to reduce the action space, tackling the challenge of large action space. Similar to the task selector, we formulate action pruning as |A| binary classification problems, and devise another world-perceiving module: the action validator. The action validator is designed to check the relevance of each action candidate a  A with respect to T t by answering questions like Is the action candidate take beef' relevant to the subtask fry chicken'?, so as to obtain a subset of actions A t  A with irrelevant actions filtered. Fig. 3 shows the module architecture. Similar to the task selector, we pre-train this module through question answering. Sample QAs have been shown in Fig. 1 (b). After pre-training, we deploy the agent in the complex games, and train the action selector through RL. We freeze the pre-trained modules, as no human labeled data will be obtained in this phase. At each time step, we use the task selector and the action validator to produce T t and A t , respectively. We keep using the same subtask T over time until it is not included in T t , as we do not want the agent to switch subtasks too frequently. The agent can simply treat T t as the additional observation of o t . If we do not limit the use of human knowledge in this phase, we can also treat T t as a goal with either hand-crafted (Jiang et al., 2019) or learnt reward function (Colas et al., 2020). Arbitrary methods can be used for optimizing (Ammanabrolu and Hausknecht, 2020; Adhikari et al., 2020). One issue we are concerned about is the compound error  the prediction error from imperfect pre-trained modules will adversely affect RL training (Talvitie, 2014; Racanire et al., 2017). For example, the false predictions made by the binary classifier in the task selector may lead to a wrong T t , which affects A t and a t in turn. To alleviate the influence of the compound error, we assign time-awareness to subtasks. A subtask is bounded 542 Table 1: Game statistics. by a time limit [0 ,  ] . If the current subtask T is not finished within its time limit, we force the agent to re-select a new subtask T t  T t \ { T } , regardless whether T is still available. Besides making the agent robust against errors, another benefit by introducing time-awareness to subtasks is that it improves the subtask selection diversity, which helps the agent to avoid getting stuck in local minima (Pong et al., 2020; Campero et al., 2020). We conduct experiments on cooking games provided by the rl.0.2 game set  and the FTWP game set  , which share the vocabulary set. Based on the number of subtasks, which is highly correlated to the number of ingredients & preparing requirements, we design three game sets with varying complexities: 3488 simple games, 280 medium games and 420 hard games. Note that there is no overlapping games between the simple set and the medium / hard game sets. Table 1 shows the game statistics. Besides Traj.Length, which denotes the average length of the expert demonstrations per game  , other statistic metrics are averaged per time step per game (e.g., #Subtasks and #Avail.Subtasks denote the average number of subtask candidates T , and the average number of available subtasks T t , respectively). We will collect human interaction data from the simple games for pre-training. We regard both medium & hard games as complex, and will conduct reinforcement learning on these two game sets without labeled data. We consider the following four models, and compare with more variants in ablation studies: IL (Chen et al., 2021): a hierarchical agent which also uses two training phases. In the first phase, both the task selector and the action selector are pre-trained through imitation learning. Then in the second phase, the action selector is fine-tuned through reinforcement learning. IL w/o FT : a variant of the IL baseline, where only the imitation pre-training phase is conducted, and there's no RL fine-tuning. Model architecture All models are implemented based on GATA's released code  . In particular, we use the version GATA-GTF, which takes only the KG-based observation, and denote it as GATA for simplicity. The observation encoder is implemented based on the Relational Graph Convolutional Networks (R-GCNs) (Schlichtkrull et al., 2018) by taking into account both nodes and edges. Both the task encoder and the action encoder are implemented based on a single transformer block with single head (Vaswani et al., 2017) to encode short texts. The binary classifier, the task scorer and the action scorer are linear layers. The GATA and IL models are equipped with similar modules. Please refer to Appendix C for details. Pre-training We train the task selector and the action validator separately, as they use different types of QAs. We ask human players to play the simple games, and answer the yes-or-no questions based on the observations. The details of the dataset construction (interaction data collection, question generation, answer annotation, etc. ) could be found at Appendix B. We train the task selector with a batch size of 256, and the action  https://github.com/xingdi-eric-yuan/ GATA-public 543 Table 2: The testing performance at 20% / 100% of the reinforcement learning phase. validator with a batch size of 64. The modules are trained for 10-20 epochs using Focal loss and Adam optimizer with a learning rate of 0.001. Reinforcement learning We consider the medium game set and hard game set as different experiments. We split the medium game set into 200 training games / 40 validation games / 40 testing games, and the hard game set into 300 / 60 / 60. We follow the default setting of (Adhikari et al., 2020) to conduct reinforcement learning. We set the step limit of an episode as 50 for training and 100 for validation / testing. We set the subtask time limit  = 5 . For each episode, we sample a game from the training set to interact with. We train the models for 100,000 episodes. The models are optimized via Double DQN (epsilon decays from 1.0 to 0.1 in 20,000 episodes, Adam optimizer with a learning rate of 0.001) with Pritorized Experience Replay (replay buffer size 500,000). For every 1,000 training episodes, we validate the model and report the testing performance. We measure the models through their RL testing performance. We denote a game's score as the episodic sum of rewards without discount. As different games may have different maximum available scores, we report the normalized score, which is defined as the collected score normalized by the maximum score for a game. Fig. 4 shows the RL testing performance with respect to the training episodes. Table 2 shows the testing performance after 20,000 training episodes (20%) / at the end of RL training (100%). Compared with GATA, which needs to be trained from scratch, the proposed QWA model achieves high sample efficiency: it reaches convergence with high performance before 20% of the training stage, Figure 4: The RL testing performance w.r.t. training episodes. saving 80% of the online interaction data in complex games. The effectiveness of pre-training can also be observed from the variant IL w/o FT: even though it requires no further training on the medium / hard games, it achieves comparable performance to our model. However, the performance of QWA can be further improved through RL, while it does not work for the IL-based model, as we can observe the performance of IL becomes unstable and drops significantly during the RL fine-tuning. A possible reason is that there exists large domain gap between simple and medium (hard) games, and our model is more robust against such domain shifts. For example, our world-perceiving task selector performs better than IL-based task selector in handling more complex observations (according to Table 1, the observations in medium / hard games contain more triplets, rooms and objects), facilitating the training of the action selector. Besides the domain gap in terms of the observation space, there is also a gap between domains in terms of the number of available subtasks  while there's always one available subtask per time step in simple games, the model will face more available subtasks in the medium / hard games. Different from our task selector, which is trained to check the availability of every subtask candidate, the IL pre-trained task selector can not adapt well in this situation, as it is trained to find the unique subtask and ignore the other subtask candidates despite whether they are also available. We further investigate the generalization performance of our model on simple games, considering that simple games are not engaged in our RL training. To conduct the experiment, after RL training, we deploy all models on a set of 140 held-out sim-544 Table 3: The RL testing performance on simple games. ple games for RL interaction. Table 3 shows the results, where Medium 100% (Hard 100%) denotes that the model is trained on medium (hard) games for the whole RL phase. The generalizability of GATA, which is trained purely with medium and hard games, is significantly low and cannot perform well on simple games. In contrast, our model performs very well and achieves over 80% of the scores. The world-perceiving modules, which are pre-trained with simple games, help to train a decision module that adapts well on unseen games. It is not surprising that the variant IL w/o FT also performs well on simple games, since they are only pre-trained with simple games. However, as indicated by the performance of IL, after fine-tuning on medium/hard games (recalling Sec. 6.1), the action scorer forgets the experience/skills dealing with simple games and the model fails to generalize on unseen simple games. In summary, the best performance achieved by QWA demonstrates that our model can generalize well on games with different complexities. We study the contribution of the subtask time-awareness by comparing our full model with the variant without this technique. Fig. 5 shows the result. Although the models perform similarly in the medium games, the full model shows better performance in the hard games, where there may exist more difficult subtasks (we regard a subtask more difficult if it requires more actions to be completed). Assigning each subtask a time limit prevents the agent from pursuing a too difficult subtask, and improves subtask diversity by encouraging the agent to try different subtasks. Besides, it prevents the agent from being stuck in a wrong subtask, making the agent more robust to the compound error. We then investigate the performance upper bound of our method by comparing our model to variants with oracle world-perceiving modules. Fig. 6 shows the results, where +expTS (+expAV) denotes that the model uses an expert task selector (action validator). There's still space to improve the Figure 5: The performance of our model and the variant without time-awareness. pre-trained modules. The variant QWA +expTS +expAV solves all the medium games and achieves nearly 80% of the scores in hard games, showing the potential of introducing world-perceiving modules in facilitating RL. We also find that assigning either the expert task selector or the expert action validator helps to improve the performance. In light of these findings, we will consider more powerful pre-training methods as a future direction. Although we only collect labeled data from the simple games, it is still burdensome for human players to go through the games and answer the questions. We are thus interested in investigating how the performance of our QWA (or world-perceiving modules) varies with respect to a reduced amount of pre-training data. Fig. 7 shows the results, where the pre-training dataset has been reduced to 75%, 50% and 25%, respectively. Our model still performs well when the pre-training data is reduced to 75% and 50%. When we only use 25% of the pre-training data, the model exhibits instability during the learning of hard games. Being pre-trained on a largely-reduced dataset, the world-perceiving modules might be more likely to make wrong predictions with the progress of RL training, leading to the performance fluctuation. However, the fi-545 Figure 7: The performance of our model with varying amounts of pre-training data. nal performance of this variant is still comparable. To summarize, our model is robust to limited pretraining data and largely alleviates the burden of human annotations. In this paper, we addressed the challenges of low sample efficiency and large action space for deep reinforcement learning in solving text-based games. We introduced the world-perceiving modules, which are capable of automatic task decomposition and action pruning through answering questions about the environment. We proposed a two-phase training framework, which decouples the language learning from the reinforcement learning. Experimental results show that our method achieves improved performance with high sample efficiency. Besides, it shows robustness against compound error and limited pre-training data. Regarding the future work, we would like to further improve the pre-training performance by introducing contrastive learning objective (You et al., 2020) and KG-based data augmentation (Zhao et al., 2021). This work is supported in part by ARC DP21010347, ARC DP180100966 and Facebook Research. Joey Tianyi Zhou is supported by A*STAR SERC Central Research Fund (Use-inspired Basic Research). We thank the anonymous reviewers for their constructive suggestions. We thank Smashicons and Trazobanana for providing the icons in Fig. 1.
abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain abstain method method abstain result abstain abstain objective method abstain method method method abstain result objective other abstain other other other other other other other objective abstain abstain abstain objective other objective method abstain other other objective objective method method other other objective method other other other other other other other other abstain objective method other other other other objective other other other method objective method abstain method abstain method method abstain abstain method method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain method abstain abstain method abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective result abstain abstain other other other other