The recent success of reinforcement learning (RL) in solving complex tasks is often attributed to its capacity to explore and exploit an environment. Sample efficiency is usually not an issue for tasks with cheap simulators to sample data online. On the other hand, Task-oriented Dialogues (ToD) are usually learnt from offline data collected using human demonstrations. Collecting diverse demonstrations and annotating them is expensive. Unfortunately, RL policy trained on off-policy data are prone to issues of bias and generalization, which are further exacerbated by stochasticity in human response and non-markovian nature of annotated belief state of a dialogue management system. To this end, we propose a batch-RL framework for ToD policy learning: Causal-aware Safe Policy Improvement (CASPI). CASPI includes a mechanism to learn fine-grained reward that captures intention behind human response and also offers guarantee on dialogue policy's performance against a baseline. We demonstrate the effectiveness of this framework on end-to-end dialogue task of the Multiwoz2.0 dataset. The proposed method outperforms the current state of the art. Further more we demonstrate sample efficiency, where our method trained only on 20% of the data, are comparable to current state of the art method trained on 100% data on two out of there evaluation metrics. Offline task-oriented dialogue (ToD) systems involves solving disparate tasks of belief states tracking, dialogue policy management, and response generation. Of these tasks, in this work we focus on dialogue policy management to improve the end-to-end performance of ToD. The need for sample  Contributed to this work during his time at Salesforce ResearchCode: https://github.com/salesforce/CASPI efficiency is key for learning offline task-oriented dialogue system, as access to data are finite and expensive. Recent advancements in off-policy reinforcement learning methods that uses offline data as against a simulator has proven to be sample efficient (Thomas and Brunskill, 2016). The effective use of these techniques are hindered by the nature of ToD. For instance, bias correction in off-policy based methods usually requires estimation of behaviour policy for a given state of Markov Decision Process (MDP). In ToD, per-turn annotated belief-state does not capture the true state of the MDP. Example of such annotated belief-state are shown in Fig:1. Latent state information such as prosody, richness of natural language and among others induces stochasticity in the agents response. In addition to these short comings, the direct use of automatic evaluation metric as reward for policy learning is not desirable, since these automatic evaluation metrics are often for the entire dialogue and not per turn. Hence such rewards are sparse and under-specified (Wang et al., 2020). Use of under-specified reward will often lead to policy that suffers from high variance (Agarwal et al., 2019). Alternatively use of imitation learning based methods falls short of reasoning on the outcome. This is demonstrated in Fig:1. Turns#3 and #2 are rich in semantic information and Turn#3 is key to success of the booking process. While Turn#4 contributes least to successful outcome. Though the turns have varying levels of importance, each of the turns are treated equally in imitation learning. In worst case, turns like Turn#4 will appear more often than turns Turn#2 and #3 in a ToD dataset, there by taking greater share of the gradient budget. We address aforementioned shortcomings with following key contributions: 1.We introduce pairwise causal reward learning to learn fine grained per turn reward that reason the intention of human utterance. for task oriented dialogue setting that guarantees performance against a baseline. With the release of multi-domain, multi-turn Multi-Woz2.0 dataset (Budzianowski et al., 2018a), there has been flurry of recent works, of which Zhang et al. (2019) uses data augmentation. Rastogi et al. (2019) and Hosseini-Asl et al. (2020) frame dialogue policy learning as language modeling task. Among the works that uses reinforcement learning. Mehri et al. (2019) uses supervised learning to bootstrap followed by RL fine tuning, whereas Zhao et al. (2019) uses policy gradient on latent action space as against handcrafted ones. Jaques et al. (2019) and Wang et al. (2020) uses Batch-RL for dialogue policy learning. (Wang et al., 2020) is first to argue the use of automated evaluation metrics directly as reward is under-specified for ToD policy learning. Recently there's has been proliferation in use of large pretrained language model based systems like Hosseini-Asl et al. (2020), Lin et al. (2020), Chen et al. (2019) etc. More details on contrasting the merits and limitations of these methods can be found in Sec:A.1 The line of inverse RL used in this work can be traced back to Ziebart et al. (2008), which proposes roll-outs from expert demonstration should have rewards exponentially higher than any other arbitrary roll-outs. This method requires a normalizing constant that integrates across rollouts, which is challenging. Christiano et al. (2017) and Thanan-jeyan et al. (2020) propose to do relative comparison of two roll-outs there by eliminating the need for normalization constant and they demonstrate in online setting. We model task-oriented dialogue as a Markov decision process (MDP) (Sutton and Barto, 2018) with set of states S and actions A . The agent at time step t with state s t performs a composite action a t as per a target policy  e ( a t | s t ) on the environment. The environment is defined by transition probabilities P ( s t +1 | s t , a t ) , a latent reward function, R ( s t , a t , g ) , discount factor   [0 , 1] and goal of dialogue g . Then the objective of the target policy  e , is to maximizes the discounted sum of future reward on the MDP, given by the state-action value function Q  e ( a t , s t ) = E a t   e ,s t  P [ (cid:80) Tt  = t  t  t  R ( s t  , a t  , g )] . In offline Batch-RL. The agent does not get to interact with the environment, instead we are provided with offline data D logged by human agents performing actions based on a latent stochastic behaviour policy  b . Rollout of a dialogue  i  D is composed of  i = (( o i 0 , a i 0 ) , ..., ( o iT  1 , a iT  1 )) . Here o t is the observation at turn t , composing of o t = ( b t , u ut , u at  1 ) , where b t is the belief state of the agent at turn t , u ut and u at  1 are the user and agent utterance at time t and t  1 respectively. Batch-RL entails training target policy  e on rollout generated by a latent behaviour policy  b . Directly optimizing on the rollouts generated by policy other than the target policy, will lead to large bias in the value function estimation, poor generalization characteristic, and sample inefficiency (Thomas and Brunskill, 2016). Safe policy improvement ensures the new policy performance is bounded by performance against a baseline policy. This is expressed as: P r ( V  e  V  b   )  1  , where V  e and V  b are value functions of the target and behaviour policy respectively. Here 1   and  are the high probability and approximation meta-parameters respectively. Schulman et al. (2015) 93 Figure 2: Shows stochacity i.e number of different dialogue act against each delexicalized belief state in Mul-tiWoz2.0 dataset provide such update mechanism, (1), whose errors are bounded as long as the constraints of (1) are met, where DKL ( . || . ) is the KL divergence and  is a hyper-parameter. L sto (  ) = min s t  P bs a t   bs  E (cid:20)  e ( a t | s t ;  )  bs ( a t | s t ) Q  bs ( s t , a t ) (cid:21) s.t. E s t  P bs [ DKL (  bs ( . | s t ) ||  e ( . | s t ))]   (1) (Schulman et al., 2015) originally formulated (1) for online learning as trust region for policy updates and uses policy before gradient update as the baseline policy,  bs ( a t | b t ;  old ) . In this work we adapt it to offline setting and use behaviour policy  b as the baseline policy. Use of this update rule requires access to the behavior policy  b ( a t | s t ) which is intractable to estimate and the learnt ones might have bias. Use of such behavior policy to perform bias correction by Important Sampling (Precup, 2000) might lead to worse policy. Instead we estimate the behaviour policy conditioned only the annotated belief-state b t as against true state s t in (1), which result in a stochastic behavior policy. This stochasticity of dialogue act vis--vis annotated belief state can observed in Fig:2. We also estimate the Q-function of the behavior policy, Q  b ( b t , a t ) using learnt reward R ( s t , a t , g ) . More on learnt reward in Sec: 3.3. The belief state b t is part of the observation o t , hence we purport that, on availability of more evidence of the observation o t , (beside b t ) the mode of the policy collapse to a near deterministic action. To factor this into the policy learning, we have an additional loss: L det (  ) = min  E ( o t ,a t )  D [ G ( , t ) log  e ( a t | o t ;  )] (2) where return G ( , t ) = (cid:80) Tt  = t  t   t R ( s t  , a t  , g ) is the discounted sum of future reward for rollout  with goal g . Hence policy optimization loss function is given by: L (  ) = L sto (  ) + (1   ) L det (  ) (3) We achieve this by doing two forward passes of the policy network  e ( a t | o t ;  ) , first with only the belief state, b t as the input and second pass with entire observation i,e o t := ( b t , u ut , u at ) as input to the policy network. We then use the corresponding action distribution  e ( a t | b t ;  ) and  e ( a t | o t ;  ) in loss functions (1) and (2) respectively. 3.3 Pairwise causal reward learning Algorithm 1 CASPI Input : Dialogue dataset D and evaluation metric M ( . ) Sub-sample K-folds of train and val set { ( DT , DV ) 1 , ..., ( DT , DV ) k | ( DT , DV )  D } for  ( DT , DV ) do Learn ToD in supervised setting by optimizing for objective:  min E a t ,s t  DT log(  m ( a t | s t )) for  epoch do Using  m ( a t | s t ) predict actions on the valset DV and add it to the dataset, DP along with corresponding metric score M (  ) for pairwise causal reward learning DP = DP  ( , M (  )) |    m end for end for repeat Sample pair of rollouts (  1 ,  2 )  DP Learn for R ( . ) by optimizing for objective (4) until Convergence using data DP repeat Optimize for policy  e using objective (3) until Convergence using data D The policy optimization objective introduced in the previous section requires access to per timestep reward R ( s t , a t , g ) . To this end, we provide a 94 Figure 3: Process flow of pairwise causal reward learning mechanism to learn a reward that is causally reasoned on the intention of the human demonstrator. Usually ToD are evaluated using dialogue level automatic evaluation metrics M ( . ) . Given the large state-action space of the dialogue management system, these dialogue level feedback are under-specified for for effective policy learning (Wang et al., 2020). Details about the the choice of evaluation metric M ( . ) are covered in Sec:4.4.2. To address this under-specified feedback, we adapt preference learning introduced by (Chris-tiano et al., 2017) from an online to an offline setting, to learn fine grained per dialogue turn (ie. per timestep t ) reward, R ( s t , a t , g ) . Given a pair of rollouts  1 ,  2  D with actions for each state in the rollout is sampled from a pair of different policies  1 m and  2 m respectively. Let  1   2 represent preference of rollout  1 over rollout  2 . This preference is true when sum of rewards of each dialogue turn of the two rollouts satisfies: (cid:80) Tt =0 R ( s t , a t , g | ( s t , a t )   1 ) > (cid:80) Tt =0 R ( s t , a t , g | ( s t , a t )   2 ) . For brevity, henceforth we refer (cid:80) Tt =0 R ( s t , a t , g | ( s T , a t )   ) as R (  ) . Then preferential probability of one rollout over an another, can be represented by: P [  1   2 ] =  ( R (  1 ))  ( R (  1 )) +  ( R (  2 )) Here  ( . ) could either be exp ( . ) or identity 1 ( . ) . In our experiments, the later works best. We optimize for reward, R ( s t , a t , g ) by minimizing binary cross-entropy loss between the preference probability and the normalized metrics score,  (  ) between a pair of rollout. We observe that the dialogue roll-outs are generated by expert latent policy. The data (dialogue rollouts) are distributed as per the optimal latent policy and transition probability. We propose that predictions made by a policy while in the process of learning to maximize the likelihood of the data is a good curriculum for exploring the state-action space for pairwise reward learning. This is a key insight of this work. We formalize this insight into a method depicted in Fig:3 and Algo:1. The (train) dataset is subsampled into K -fold train & val sets. K -baseline policies are trained to fit the data distribution generated by experts using cross entropy loss, i.e supervised learning. During the process of fitting the data distribution, the still learning K-policies are used to predict on their corresponding K-fold valset at every epoch of the training. Each of these predictions are the scored by a chosen dialogue level metric, M ( . ) . On convergence of this supervised learning process, pairs of dialogue predictions generated by the above process, along with their corresponding metric score are used to train for fine grained reward R ( a t , s t , g ) using objective (4). The use of K-fold subsampling, K-baseline policies, .m and actions sampled from these K-policies that are still in the process of learning help generate counter factual examples in the action space. These counter factual actions close to optimal policy, along with the goal of the dialogue helps us to learn subtle nuance of fine grained reward function R ( a t , s t , g ) in the region of action space that matters the most. 4 Experimental Settings 4.1 Model 4.1.1 CASPI(.) The learnt reward using CASPI R ( s t , a t , g ) is akin to sample weights for each dialogue turn, that helps to redistribute the gradient budget among dialogue turns based of their contribution to the overall success of the ToD. Hence we believe our pairwise casual reward learning and associated improvement in sample efficiency are independent of model architecture. To this end we choose two ToD methods that are at the extremes of model architecture spectrum 1) One uses a light weight custom model and 2) Other uses a large standard pre-trained out-of-the box universal language model. In this setting , we use the neural model proposed by Zhang et al. (2019). DAMD is composed of three seq 2 seq generative model using GRUs. The three seq 2 seq models are one each for belief state, dialogue act and response generation modules. An attention layers is used to attend the outputs of the seq 2 seq models with the context vector of previous turn for copy over mechanism. The outputs of these attention layer are used as representation for predicting series of tokens for their respective modules. For more details on the model architecture and parameter setting refer Zhang et al. (2019). In this setting we use both stochastic, L sto and deterministic, L det loss functions on dialogue act. For DST and response generation, we retain the cross entropy loss as is from DAMD (Zhang et al., 2019). 4.1.3 CASPI(MinTL) On the other extreme of model complexity, we use the Task oriented Dialogue model, MinTL(Lin et al., 2020). MinTL uses a large pretrained language model BART (Lewis et al., 2019). BART use as a standard encoder decoder transformer architecture with a bidirectional encoder and an autoregressive decoder. It is pre-trained on the task of denoising corrupt documents. BART is trained using cross-entropy loss between the decoder output and the original document. For more details of the model architecture and parameter setting, we suggest referring to (Lin et al., 2020) (Lewis et al., 2019). MinTL doesn't explicitly predict dialogue act. Hence we only use the deterministic loss, L det directly on the generated response and for DST we retain the loss as is from MintTL (Lin et al., 2020). For k-model training of pairwise casual reward learning illustrated in Fig:3, we chose DAMD (Zhang et al., 2019) model for it's light weight model architecture. In all our experiments, we use K = 10 . For the pairwise casual reward learning network, we use three single bi-LSTM layers, one each to encode goal, belief state and either dialogue act or response sequences at each dialogue turn on each of the sampled roll-outs pairs,  1 and  2 . The three encoded representations are concatenate and are fed through a couple of feed-forward layers before making a bounded reward prediction R ( s t , a t , g )  [0 , 1] for each turn using a sigmoid function. The per turn rewards are summed to form a global reward R (  ) for the roll-out  . Using a pair of dialogue rewards R (  1 ) and R (  2 ) , we compute the probabilistic preference between the roll-outs P [  1   2 ] either by standard normalization or a softmax function. The output of this optimized using binary crossentopy loss described in Eqn:4. The above described architecture is illustrated in Fig:10 . To evaluate our proposed method on Multi-domain Wizard-of-Oz (MultiWoz) (Budzianowski et al., 2018a) dataset. It is a large scale multidomain, task oriented dataset generated by human-to-human conversation , where one participant plays the role of a user while the other plays the agent.The conversations are between a tourist and a clerk at an information center. The conversations span across 7 domains including attraction, hospital, hotel, police, restaurant, taxi and train. Each dialogue is generated by users with a defined goal which may cover 1-5 domains with a maximum of 13 turns in a conversation. The dataset has 10438 dialogues split into 8438 dialogues for training set and 1000 dialogues each for validation and test set. We represent DB results as one-hot vectors as proposed by Budzianowski et al. (2018b). To reduce surface-level variability in the responses, we use domain-adaptive delexicalization preprocess-96 ing proposed in Wen et al. (2016). As proposed in Zhang et al. (2019), We generate delexicalized responses with placeholders for specific values which can be filled with information in DST and database. We evaluate performance of our method on end-to-end dialogue modeling task of Multi-woz2.0 (Budzianowski et al., 2018a). We uses three evaluations metrics proposed by (Budzianowski et al., 2018a). These include: 1) inform rate measures the fraction of dialogue, the system has provided the correct entity, 2) success rate fraction of dialogues, the system has answered all the requested information and 3) BLEU (Papineni et al., 2002) measures the fluency of the generated response. We also report the combined score ( Inform + Success )  0 . 5 + BLEU proposed by Mehri et al. (2019). All the numbers of CASPI reported in this work are median of 5 runs with different seeds. For the metric M used in pairwise causal reward learning , we use the following: This is very similar to combined score used in evaluation and both are equivalent when  = 2 . We introduced hyperparamter  to normalize the achievable scale of BLEU . We observe that success rate, if used as is, will result in non-markovian and stochastic per turn reward function. This is because the reward of current state will depend on the performance of future states. Hence, we also use a soft version of the metric M soft , where the success rate measures a fraction of requested information provided in a dialogue. We refer the original metric that uses the discrete variant of success rate as M hard . The choice of action in reward function R ( s t , a t , g ) can either be dialogue act or generate response, we refer corresponding variants of metrics as M ( act ) and M ( resp ) . To demonstrate the versatility of our method to adapt to different metrics, we use all the discussed variants of the metric. 5 Result We compare both adaptation of our methods CASPI(DAMD) and CASPI(MinTL) on the end-to-end dialogue tasks defined by MultiWoz2.0 (Budzianowski et al., 2018a). The results are tabulated at Table:1. CASPI(DAMD) with its light weight model architecture and no pretraining on any external corpus, except for (Lubis et al., 2020), out perform all other previous methods, these includes methods that use large pretrained language models such as Hosseini-Asl et al. (2020), Peng et al. (2020) and Lin et al. (2020). This show using CASPI to shepard the gradient update process as sample weights for each dialogue turn leads to a model that's well aligned with true objective of the task. CASPI(MinTL) with its robust pretrained model out performs CASPI(DAMD) and LAVA (Lubis et al., 2020) by a large margin. This demonstrates the ease of adaptation of existing methods with CASPI. Inverse reinforcement learning, coupled with off-policy policy learning and evaluation are proven to be sample efficient (Thomas and Brunskill, 2016) . We argue CASPI is competitive with other sample efficiency techniques, such as data augmentation and transfer learning as performed by Zhang et al. (2019) and Lin et al. (2020) respectively. To demonstrate the hypothesis, we test our method against baseline in a low sample complexity regime. For experimental setup, we adopt the low resource testing strategy from Lin et al. (2020). We train our model on 5%, 10%, and 20% of the training data and compared with other baselines on end-to-end dialogue task, Table 2 list the results. CASPI(MinTL) trained only on 20% of data was able to out perform previous state of the art method, LAVA (Lubis et al., 2020) and MINTL (Lin et al., 2020) trained on 100% data on two of the three performance metrics. This goes to show that having the right reward function to guide the budget of the gradient update process to reach the true objective is important in extremely low resource setting. Automatic evaluation metrics have their own biases. True objective of ToD is human experience while interacting with the dialogue systems, which automatic evaluation metrics might fall short to capture. To this end we conduct human evaluation on the quality of the generated response. We define quality by the following criterias: A dialogue turn in the test set is randomly picked. The human evaluators were shown context leading up to the turn. The predictions for the turn by different methods were anonymized and displayed to the evaluators. This is illustrated in Fig:4. The human evaluators were asked to give a score between 1 and 5 for appropriateness and fluency, with score of 5 being best and 1 being the worst. 100 randomly selected dialogue turns were presented to 10 participants .We report the mean and variance of the score. We compare our model performance against MinTL (Lin et al., 2020), SimpleTOD (Hosseini-Asl et al., 2020), LAVA (Lubis et al., 2020) and DAMD (Zhang et al., 2019). Fig:5 shows the results of the evaluation. CASPI(MinTL) outperforms all other models in appropriateness score. While fluency score of CASPI(MinTL), MinTL and SimpleTOD are comparable to each other. It is worth noting that though LAVA (Lubis et al., 2020) performs well on automatic evaluation metrics, it performs poorly on human evaluation. We suspect the policy learnt by (Lubis et al., 2020) exploits gaps in the reward function. In case of LAVA (Lu-bis et al., 2020), success rate is used as the reward function. In our analysis, low BLEU score is good indicator if the learnt policy indulges in reward hacking, which LAVA (Lubis et al., 2020) exhibits. More on reward hacking in Sec:5.4.2. In the previous section we argued that automatic dialogue evaluation metrics are biased and doesn't truly reflect the human objective, but in our method we use these very same dialogue evaluation metrics to learn reward R ( s t , a t , g ) . To bridge this gap, we performed the following human-in-the-loop (HITL) experiment. We first trained a pair CASPI(MINTL) models with different seeds, on 5% of Multiwoz2.0 dataset. We then used these pair of models to predict on 0.5% of Multiwoz2.0 train data (40 dialogues) and had a human score these pairs of generated response relative to each other. We then trained for reward R ( s t , a t , g ) using pairwise causal reward learning as described in Sec:3.3, where examples of the mini batch are randomly sampled either from human scored examples or the ones scored by the automatic evaluation metric as show in Fig:6. We then trained a fresh CASPI(MINTL) model on the original 5% of data and the learnt R ( s t , a t , g ) . We perform human evaluation on 24 dialogues using 3 participants. Fig:7 98 Figure 4: Example of generated responses by different ToD models Figure 5: Human evaluation on crite-rias:Appropriateness and Fluency shows the performance. Though CASPI(MINTL) using just 5% of the data outperforms DAMD trained on 100% of data in 2 out of the 3 automatic evaluation metrics shown in Table:1 and 2, performs poorly in human appropriateness score. With the HITL score in the reward learning, we see a boost in performance in both the human evaluation criteria: appropriateness and fluency. The 5% data CASPI(MINTL)'s human approriateness score is now comparable to 100% data DAMD. This goes to show the versatility of the pairwise causal reward learning. With enough expressiveness of the neural network used, the pairwise causal reward learning can generalize to unknown dialogue evaluation criteria. In this section we qualitatively analyze the results of pairwise causal reward learning. Fig:8 is the same conversation between a tourist and information center agents that we introduced earlier, now we have learnt reward R ( s t , a t , g ) , against each turn. We observe that Turn#3 has received the highest reward, retrospectively we realize the transFigure 6: Mixed Human-in-the-loop and automatic evaluation metric scores for pairwise causal reward learning Figure 7: Human evaluation of Human in the loop training of CASPI(MinTL) on 5% of Multiwoz2.0 dataset action happens in this turn, which is crucial and has to be risk averse for the success of the dialogue. Turn#2 gets the next best reward which captures crucial information needed for transaction to happen in Turn#3. Turn#4 gets reward an order lower than Turn#3 & 2 because other than nicety, it doesn't contribute much to the success of the conversation. It should be noted that responses like Turn#4 will appear in almost all conversations and in supervised learning, these turns will be receiving the highest share of the gradient budget. The learnt reward redistributes the gradient budget based on the turns contribution to the success of the dialogue objective. In this section we analyze the type of behaviour CASPI agents sometime exhibit, especially when trained in low sample regime. Greedy agent: In certain domains, the agents has a tendency to book a service before it has gathered all the required information or before the user requested or agreed for booking a service. The first example in Fig:9 demonstrate this behaviour. Here the user has requested for a taxi, before enough information such as destination or time of departure 99 Figure 8: Example of learnt reward Figure 9: Example of agent behaviour in low sample regime. are gathered, the agent books the taxi. This happens because there are gaps in automatic evaluation metrics. A low BLEU score and relatively high inform and success rate might indicate greedy agent behaviour. Other reasons for low BLEU score includes: lack of diversity in the responses or malformation of response. Cautious agent: The agent tends to be cautious by providing long winded replies packed with more information than needed. Agent tend to do this to prevent the risk of loosing rewards by missing out any requested information. This behaviour is demonstrated in the second example in Fig:9 These subtle behaviour demonstrates gap in automatic evaluation metrics, which could be weeded out using Human in the loop learning described in Sec:5.3. In this work we introduced a fine grained reward learning process using an under-specified metrics and expert demonstrations for efficiently learn task oriented dialogue. We demonstrated the efficacy of our method on MultiWoz2.0 dataset with results comparable to the existing state of the art method with only 20% of data. We believe the methods is generic and can be extend to other NLP tasks.
abstain abstain abstain abstain abstain objective abstain objective abstain objective abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain other other other other other other other other abstain other other method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain method abstain method abstain method method method method other method abstain abstain abstain method abstain abstain abstain method abstain method method method abstain method method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective objective