Conditional text generation often requires lexical constraints, i.e., which words should or shouldn't be included in the output text. While the dominant recipe for conditional text generation has been large-scale pretrained language models that are finetuned on the task-specific training data, such models do not learn to follow the underlying constraints reliably, even when supervised with large amounts of task-specific examples. We propose NEUROLOGICDECODING , a simple yet effective algorithm that enables neural language models  supervised or not  to generate fluent text while satisfying complex lexical constraints. Our approach is powerful yet efficient. It handles any set of lexical constraints that is expressible under predicate logic, while its asymptotic runtime is equivalent to conventional beam search. Empirical results on four benchmarks show that NEUROLOGICDECODING outperforms previous approaches, including algorithms that handle a subset of our constraints. Moreover, we find that unsupervised models with NEUROLOGICDECODING often outperform supervised models with conventional decoding, even when the latter is based on considerably larger networks. Our results suggest the limit of large-scale neural networks for fine-grained controllable generation and the promise of inference-time algorithms. Text generation applications often need to incorporate semantic constraints, i.e., what words should and shouldn't appear in the output generation. Consider the task of generating a recipe from a set of ingredients (Kiddon et al., 2016), such as garlic,' steak', and soy sauce' (Figure 1). A generated recipe should cover all of those ingredients, without hallucinating new ones (such as pork' or beans'). This restriction, like others in Figure 1 for other i npu t Scenario food | table | sit | front The man sat with his food at the front of the table The food is in front of you sit at the table. constraints expressed as a predicate logic formula. The dominant paradigm today for performing such constrained generation is to start with a pretrained language model, and then finetune it on a dataset of task-specific examples. However, pretrained language models struggle at learning to follow these constraints, even when the finetuning dataset is large. For example, for the aforementioned recipe generation task, a GPT2 model finetuned on hundreds of thousands of recipes still hallucinates extra ingredients. In stark contrast, humans need to see only a few examples (or even none) to generate the desired output satisfying all the logical constraints, e.g., writing a recipe that mentions each ingredient (butter, steak, etc.) without using new ones. We hypothesize that this mismatch is due to a fundamental under-specification of finetuning. If we finetune one of today's state-of-the-art language models on a dataset, the likelihood of it generating sequences from the same distribution should increase. Yet there is no guarantee that this improvement in likelihood will come from improvements on the fundamental task of constrained generation, as opposed to picking up on dataset-specific patterns such as language style. In fact, we present analysis suggesting that worst-case' learning behavior is common in practice: when we increase the finetuning data fed to GPT2 by an order of magnitude, constraint-satisfaction with standard beam search shows only modest improvement. To address this issue, we propose NEUROLOGICDECODING , which effectively enforces the satisfaction of given lexical constraints by controlling the decoding stage of sequence generation. These constraints can be any predicate logic formula, which crucially includes both positive constraints (the word butter' must be generated somewhere) and negative constraints (bean' cannot be generated). These simpler constraints can then be combined through logical connectives to handle more complex requirements such as inflection or synonyms (beef' or steak' both satisfy the constraint of referring to the steak). While beam search aims to maximize the likelihood of the generated sequence, our method searches for optimal output sequences among the strings that also satisfy the given constraints. It does so efficiently: we convert the hard logic constraints into a soft penalty term in the decoding objective, and use a beam-based search to find approximately-optimal solutions; constraint states are tracked to reuse computation. NEUROLOGICDECODING thus effectively and efficiently controls text generation without requiring any mod-ification of the model structure or training pipeline. reasoning (COMMONGEN ; Lin et al., 2020), recipe generation (Kiddon et al., 2016), data-grounded dialogue response generation (Wen et al., 2015), and reducing gender bias in machine translation (Stanovsky et al., 2019). Empirical results demonstrate that NEUROLOGICDECODING ensures the satisfaction of given constraints while maintaining high generation quality, in turn leading to new SOTA results in both the supervised and zero-shot setting. In this section, we first rigorously define predicate logic constraint, and then present in detail the NEUROLOGICDECODING algorithm. Let us define a predicate D ( a , y ) to be a boolean function indicating the occurrence of key phrase a in a sequence y , where a can be either unigram or multi-gram. D ( a , y ) will be true iff a occurs in y . NEUROLOGIC accepts lexical constraints in Conjunctive Normal Form (CNF): where each D i represents a single positive or negative constraint, D ( a i , y ) or  D ( a i , y ) , restricting whether key phrase a i should be strictly included or omitted in y , respectively. Any propositional logical formula can be converted to CNF, and thus handled by NEUROLOGIC . Notationally, we will refer to each individual constraint D i as a literal , and the disjunction of literals as a clause , denoted as C j , with L being the total number of clauses. Our method seeks optimal sequences in which all clauses are satisfied:  y = arg max y Y P  ( y | x ) where L (cid:88) i =1 C i = L (1) Past work on constrained optimization introduces penalties (Fiacco, 1976) to approximate the constrained optimization problem with an unconstrained problem. Specifically, by adding a high-cost penalty term for violated constraints:  y = arg max y Y P  ( y | x )   (cid:48) L (cid:88) i =1 (1  C i ) (2)  ! Intuitively, this objective balances sequence likelihood (term 1) and constraint satisfaction (term 2). The aim is to find sequences that do well at both dimensions. While exhaustive search is intractable, we use a beam-based search to find approximately-optimal solutions for this objective. When considering whether a generation hypothesis satisfies some clause C i during generation, there are fundamentally 4 possible states (as in figure 2) S1 reversible unsatisfaction : If an unsatisfied clause C i contains at least one positive literal, C i could be satisfied in the future by fulfilling one of its positive literal(s). S2 irreversible unsatisfaction : If an unsatisfied clause C i contains negative literal(s) only, C i will maintain unsatisfied in the future since the violation of negative literals could not be overturned. S3 reversible satisfaction : If all satisfied literal(s) in a satisfied clause C i are negative literal(s), C i could switch back to unsatisfied in the future by violating all of its satisfied negative literal(s). S4 irreversible satisfaction : If satisfied literal(s) in a satisfied clause C i contains at least one positive literal, C i will maintain satisfied in the future since the fulfilment of positive literals is irreversible. To track the states of literals and clauses efficiently, we maintain two prefix tries. The first trie, T + , tracks unsatisfied positive literals from all clauses in states S1 and S3, while the other trie, T  , tracks satisfied negative literals from all clauses in state S3. We do not track anything from clauses in state S2 or S4, as those are already irreversible. S1 or S3 is henceforth irreversibly satisfied (state S4), thus we remove all literals of that clause from both tries and stop tracking. If a negative literal in state S3 is violated, we remote it from the trie T  . Once all negative literals of a clause in state S3 has been removed, the clause switches back to unsatisfied (state S1 or S2). If it has unsatisfied positive literal(s) in the trie T + , it becomes reversibly unsatisfied (state S1); otherwise it shall stay irreversibly unsatisfied (state S2). Since exhaustive search to optimize the CNF constraints is intractable, NEUROLOGIC uses a beam-based search to approximate. The high-level intuition is that at each time step, NEUROLOGIC selects generation hypotheses in consideration of both the objective function and the diversity of the partially satisfied constraints. We achieve such by 3 steps: pruning , grouping , and selecting (illustrated in figure 3, and detailed below). At each time step, the decoding model generates a distribution over all vocabulary V for k hypotheses in the current beam, resulting in a candidate score matrix of size k | V | . Along with generating score matrix, we produce a constraint state for each of the k  | V | new candidates h , based on the next token considered. Pruning step : We first discard any h with irreversible unsatisfied clause (state S2) to focus only on candidates that might satisfy all constraints. Then, we filter candidates h to those in the top-tier of both satisfied constraints and sequence likelihood. Specifically, we drop any candidates not in the top in terms of likelihood P  ( y t | y <t ) , and not in the top in terms of number of satisfied clauses (cid:80) Li =1 C i . These are adjustable parameters, corresponding to maximum tolerance to sequence fluency and constraint satisfaction. Grouping step : Next, we select the beam from the pruned candidates. Naively selecting k best candidates with respect to the objective function would not work well, since such greedy selection would bias toward sequences with high likelihood and easy-to-satisfy clauses at early timestep, which can lead to struggling with remaining hard-to-satisfy clauses later on. Therefore, the key intuition is to consider diverse partial solutions early on with respect to the set of irreversibly satisfied clauses, i.e., { C i | C i  state S4 } . We group candidates based on this set and select (in the next step) the best ones Constraints  ! cowbo  " dog  (  # (play music)  $ plays music )  (  % catch  & catches )  #  $  !  " runs catches plays eats plays talks talks plays catches cowboy man dog The t = 0 t = 1 t = 2 search tree score 3 1 4 2 0.18 + 0.1 * 0 = 0.18 0.12 + 0.1 * 0 = 0.12 0.15 + 0.1 * 0 = 0.15 0.11 + 0.1 * ! from each group, we first rank candidates within a group by score function: where  a i is a i 's matched prefix with ongoing generation. For example, for y = The boy climbs an apple and constraint a i = apple tree , we have  a i = apple . The second term denotes maximal percentage of matched prefix in partially satisfied positive literals. Intuitively, this score function ranks candidaites by likelihood and gives a partial reward to candidates moving towards satisfying a positive literal in an unsatisfied clause (state S1). is an adjustable parameter, controlling how much we favor candidates towards fulfilling another unsatisfied clause. We then proceed in rounds of filling the beam, visiting each group and taking the best scoring ones in rotation, until we reach k candidates. The group traversing order follows the descending order of the highest score in each group. In the end, we take the hypothesis with highest likelihood from the ones with maximal satisfied clauses. NEUROLOGIC distinguishes itself from past works in constrained decoding in 3 fundamental ways. of CNF constraint, while previous works only allow a subset of this (typically conjunctions). Second, NEUROLOGIC effectively optimizes objective function through efficient and diverse search over output space, while previous works suffer from either myopic and narrow or inefficient exploration of the search space. Third, the asymptotic runtime of NEUROLOGIC is O ( Nk ) 1 , same with beam search, constant with respect to number of constraints C . Some previous works suffer from exponential runtime, making applications infeasible. A detailed comparison between NEUROLOGIC and previous methods is provided in table 1. 3.1 Previous Constrained Decoding Approach Anderson et al. (2017) propose constrained beam search ( CBS ), where constraint satisfaction is tracked by a finite-state machine with 2 C states (all possible satisfaction status for C constraints). Beam search is done over all states with k candidates per state. This method has an exponential complexity O ( Nk 2 C ) , making many applications infeasible. Hokamp and Liu (2017) propose grid beam search ( GBS ), which groups together hypotheses by number of constraints satisfied, giving C + 1 1 N denotes sequence length and k denotes beam size. In this paper, we the asymptotic runtimes is in terms of the number of calls to a deep generator that scores P  ( y t | y <t ) ; this is because calling the generator is the most expensive part of decoding (as opposed to auxiliary bookkeeping). groups altogether. Each group stores at most k candidates that are expanded at each timestep. GBS has a faster runtime of O ( Nk C ) , but this approach biases towards sequences satisfying constraints greedily, and collapses into very similar search paths that are often times globally sub-optimal, which results in dropped language quality. Post and Vilar (2018) propose dynamic beam allocation to reduce GBS's explicit dependence on C . Beam search is done over a single beam, with the k slots of this beam dynamically allocated over the C +1 groups explicitly used by GBS. This approach was made GPU-efficient by Hu et al. (2019a). Still, the language quality issue of GBS remains, and can be worse in practice as fewer hypotheses are considered at each step. Miao et al. (2019) propose Constrained Generation by Metropolis-Hastings Sampling ( CGMH ). This approach begins by inserting all positive-constraint keywords in random order. Edits are randomly sampled to replace, insert, or delete words to make the sentence fluent; the probability of each action is computed on top of a language model. Sha (2020) proposes using gradient of a objective function to guide where and how to edit instead of random sampling. These approaches have runtime independent to number of constraints; yet they can involve repeated deletions and insertions, reducing efficiency. Generation quality is also sensitive to initial keyword order and sampled edits. Lexically constrained generation can be broadly applied to prior conditional text generation tasks. Examples include incorporating pre-specified lexical constraints (Anderson et al., 2017; Post and Vilar, 2018), user-provided terminology constraints (Hasler et al., 2018; Dinu et al., 2019), noisy automatic constraints (Li et al., 2019) in translation output. A major use case of lexical constrained decoding is paraphrase generation (Hu et al., 2019a; Kajiwara, 2019; Hu et al., 2019b; Miao et al., 2019), by negatively constraining words in the source to enforce paraphrasing. Another use case is image captioning, with novel scenes or out-of-domain objects (Anderson et al., 2017), or requiring explicit grounding to objects in the scene (Ren et al., 2015; Krause et al., 2016). In addition, Balakrishnan et al. (2019) leverage constrained decoding to improve semantic correctness for response generation. COMMONGEN (Lin et al., 2020) is a benchmark dataset designed as a test of generative commonsense reasoning. Given a set of common concepts (e.g., dog, frisbee, catch, throw); the task is to generate a coherent sentence describing an everyday scenario using these concepts (e.g., a man throws a frisbee and his dog catches it). Problem Formulation The input is an unordered set of n concepts x = { a 1 , a 2 , . . . , a n } , where each concept a i is a common object (noun) or action (verb). The expected output is a simple, grammatical sentence y  Y that describes a common scenario using all given concepts in x with correct morphological inflections. To apply NEUROLOGICDECODING , we impose that each a i must appear in output y under some morphological inflection. Let  a i = {  a i 1 , . . .  a i |  a i | } denote all inflections of a i . y covers concept a i , if at least one of {  a i 1 , . . .  a i |  a i | } appears. Formally,  a i  x ,   a ij   a i , D ( a ij , y ) where D ( a ij , y ) is a boolean-value function indicating whether y contains  a ij or not, as defined above. 2 2 This gets converted into  ni =1 (cid:0)  |  a i | j =1 D ( a ij , y ) (cid:1) . Dataset The COMMONGEN dataset consists of 35,141 concept-sets (32,651 in train , 993 in val , 1,497 in test ) associated with 77,449 sentences. The average size of the concept-sets in the test set is 4 . 04 , with an average of four sentences per concept-set and an average sentence length of 13 . 34 words. Approach and Baseline The standard pipeline of approaching this problem is to consider it as a conditional sentence generation task. We experiment with several recent pre-trained language models, including GPT-2 (Radford et al., 2019), UniLM (Dong et al., 2019), UniLM-v2 (Bao et al., 2020), BERT-Gen (Bao et al., 2020), BART (Lewis et al., 2020), and T5 (Raffel et al., 2019). All models are finetuned with their default hyperparameters. We compare with commonly used decoding methods, including beam search, sampling, and also previously proposed constrained decoding methods. We use several widely-used automatic metrics to automatically assess the performance, such as BLEU, ROUGE, METEOR, which mainly focus on measuring surface similarities. We also include metrics specially designed for captioning task, such as CIDEr, and SPICE. Following Lin et al. (2020), we report the concept Coverage, which is the average percentage of input concepts that are present in lemmatizatized outputs. In Table 4, we first present comparisons across different decoding methods based on a supervised sequence-to-sequence model, GPT-2. The key observations are: 1. NEUROLOGIC outperforms all other previous decoding methods, both constrained and unconstrained, with respect to all metrics and often with a significant margin. 2. NEUROLOGIC not only attains high constraint satisfaction ( COVERAGE ), it also improves the generation quality as quantified over ROUGE , BLEU , METEOR , CIDE r, and SPICE . 3. In comparison, all previous constrained decoding methods (Hokamp and Liu, 2017; Post and Vilar, 2018; Hu et al., 2019a) attain high constraint satisfaction at the cost of generation quality; being outperformed here by conventional beam search with a large margin. The second and the third points above demonstrate that the improved logical expressiveness of NEUROLOGIC together with the effective search strat-1 Figure 4: Performance (y-axis) of supervised GPT2-L on COMMONGEN , with a varying amount of training data for supervision (x-axis). Table 2 presents experiments across various state-of-the-art pre-trained language models. In this experiment, all models are supervised on the COMMONGEN training dataset. Under each column,    shows the performance using the conventional beam search (  ) compared to the enhanced performance using NEUROLOGICDECODING (  ). As before, NEUROLOGIC always improves the performance across all models and all metrics with no exception  both in terms of constraint satisfaction as well as generation quality. The improvement is especially substantial when the generation quality is relatively low due to smaller model capability or less efficient model architecture or pre-training. In this experiment, we test how well NEUROLOGIC works with unsupervised pre-trained language models, with and without domain adaptation. Table 3 presents experimental results of zero-shot (i.e., unsupervised) constrained generation. With unconstrained decoding, we have zero controllability over the unsupervised language models, as they ignore the problem input and generate irrelevant text. With NEUROLOGIC , on the other hand, we can dramatically improve the performance on all metrics. Fig 6 demonstrates some generated examples. In zero-shot setting without any finetuning, the language style of pre-trained LMs might differ from that of COMMONGEN . To further improve the performance, we conduct language domain adaption by fine-tuning the language models on the training-set COMMONGEN language  ignoring all concept sets. We observe that after domain adaption, NEUROLOGIC in zero-shot setting outperforms unconstrained generation with supervised finetuned LMs, which suggests that inference-time algorithms can provide a more compute-efficient avenue to draw better from neural models. The amount of training data Figure 4 compares the performance (y-axis) of supervised GPT-2 with NEUROLOGIC ( orange line ) and with conventional beam search ( blue line ) as a function of the increasing amount of training data (x-axis). Notably, even after being supervised on 100% of the training data, the supervised GPT-2 does not successfully learn the COMMONGEN constraints (Coverage') and is even outperformed by the zero-shot GPT-2 (i.e., using 0% training data) with NEUROLOGIC . The model size Figure 5 compares the performance (y-axis) of GPT-2 with varying model sizes (x-axis). Regardless of the model size, NEUROLOGIC ( purple line and black line ) boosts performance considerably over conventional beam search ( blue line ). More over, if using NEUROLOGIC , the performance of unsupervised models ( black line ) becomes comparable to that of supervised models ( purple line ). Remarkably, unsupervised models with NEUROLOGIC based on smaller networks ( black line ) often outperform supervised models with conventional beam search based on considerably larger networks ( blue line ). We next study cooking recipe generation, a paragraph-level generation task. Given a dish name and a list of ingredients, the task is to generate cooking instructions for the given recipe. Problem Formulation The input is the recipe title, an unordered set of ingredients E = { e 1 , ..., e | E | } where e i can be a singleor multiword ingredient phrase (e.g., onions', black pep-per'). Let G denote the set of all ingredients. The expected output is a paragraph y  Y that describes multi-step cooking instructions. To apply NEUROLOGICDECODING , we constrain output y to contain all given ingredients e i in E , and no other ingredients, i.e. no ingredients in G \ E . Ingredients can be referred to with generic terms (e.g., vegetables' may refer to onions', or carrots') and we denote the generic name for ingredient e i as e Ti . Formally, the constraint is (cid:16)  e i  E, D ( e i , y )  D ( e Ti , y ) (cid:17)  (cid:16)  e i  G \ E,  D ( e i , y ) (cid:17) Dataset, Approach and Baseline We use Recipe1M+, a large-scale, structured corpus of over one million cooking recipes. On average each recipe has 118 words and 9 ingredients. RecipeGPT (Lee et al., 2020) is a GPT-2 model fine-tuned on Recipe1M+, for generating recipes. Its default decoding algorithms are beam search and sampling, which serve as the baselines for evaluating our method. In addition, we compare against previously proposed constrained decoding methods with RecipeGPT. Besides common evaluation metrics for generation task, we introduce explicit measures of given-ingredient coverage and usage of extra/hallucinated ingredients. Result Table 5 presents the experimental results. We can see that NEUROLOGIC outperforms all baselines in all metrics. The delta is quite remarkable on coverage of given ingredients and usage of extra ingredients. With NEUROLOGIC , we are able Supervised? to cover almost all ingredients in generated instructions and guarantee not to use any other ingredients, which leads to more accurately controlled generation. By plugging NEUROLOGIC into existing generation system, we can get immediate boosts in controllability and generation quality with no extra computational cost. In dialogue response generation for hotel and restaurant information systems (Wen et al., 2016), we generate a natural language response given a query type (e.g., informing or querying) and a list of facts to convey (e.g., a hotel's name and address). Problem Formulation The input is a query type, an unordered set of facts F = { f 1 , ..., f | F | } , where each f i contains attribute and value (i.e. accepts_credit_cards=yes, name=red victorian bed breakfast). The expected output is a dialogue responses y  Y containing given information. The constraint here is that all given facts f i must be included in responses y in proper natural language form f Ni . We use a very simple template to turn f i to natural language form f Ni . (i.e. the natural language form for accepts_credit_cards=no is doesn't accept credit cards). Formally,  f i  F, D ( f Ni , y ) Dataset, Approach and Baseline We use the hotel and restaurant dialogue system corpus and the same train-dev-test split from (Wen et al., 2016). There are 8 query types and 12 attribute types. The standard paradigm for dialogue generation is to consider it as a conditional sentence generation task and finetune a seq2seq model. While this pipeline works effectively with existing data, once we have user queries with new query types or new attribute types, the seq2seq model would not be able to generate plausible responses. The Model Accuracy (%;  )  S (F1;  ) E n -D e Google Translate 59.4 12.5 Microsoft Translator 74.1 30.2 Junczys-Dowmunt et al. 60.5  91.0 13.3  4.3 Junczys-Dowmunt et situation can happen frequently with a dialogue generation system in application. Thus, we are interested in zero-shot dialogue generation. We give a hand-crafted initial prompt to a pre-trained LM based on the query type and apply NEUROLOGICDECODING to force given facts to include in generation. The pre-trained LM we use here is GPT-2 (Radford et al., 2019). The baseline we compare against is seq2seq finetuned LMs with vanilla beam search, including GPT-2 (Radford et al., 2019), BART (Lewis et al., 2020) and T5 (Raffel et al., 2019). We also compare with previous SOTA (Kiddon et al., 2016) on dialogue response generation. Result Table 6 presents the experimental results. We can see that zero-shot generation with NEUROLOGIC outperforms or matches supervised baselines. This suggests that plugging NEUROLOGICDECODING into pretrained LMs can lead to a powerful dialogue generation system, we do not actually need massive finetuning with extra computational cost to do that. Problem Formulation We adopt the task setup and dataset of Stanovsky et al. (2019). The input x is an English sentence describing a scenario with human entities N = { n 1 , . . . , n | N | } who are iden-tified by roles. The desired output is a translation y which uses the correct gender inflections in the target language (here, German or French). through coreference resolution, linking each entity with their gendered pronoun. 3 We then constrain the correctly-gendered human entities to appear in output y . For a human entity n i , let n Fi denote its female inflection in the target language, and n Mi denotes its male inflection. Let F denotes the set of human entities associated with female characters, and M denotes the set of entities associated with male. Formally, the constraint is (cid:16)  n i  F, D ( n Fi , y )   D ( n Mi , y ) (cid:17)  (cid:16)  n i  M, D ( n Mi , y )   D ( n Fi , y ) (cid:17) Dataset We use Stanovsky et al. (2019)'s dataset, which is built over the English-only coreference gender-bias studies: Winogender (Rudinger et al., 2018) and Wino-Bias (Zhao et al., 2018). Result Our results are shown in Table 7. When provided gender markers given by a coreference model, NEUROLOGIC increases the accuracy of handling gender correctly by 30.5 percentage for German, and 28.0 percentage for French. This even outperforms commercial translation systems  the best result, over any language or system, is Microsoft Translator for German with 74.1% accuracy, whereas NEUROLOGIC enables the baseline model to get 91% accuracy. The performance increases again by an additional 4% (German) and 8.9% (French) when ground-truth gender markers are used during constrained decoding. Last, the diagnostic results also show that NEUROLOGIC is particularly effective at reducing (over)reliance on stereotypical gender roles, with a significant decrease in performance difference  S between stereotypical and non-stereotypical gender roles. These results suggest that NEUROLOGICDECODING is a plug-and-play approach for reducing gender bias in existing translation systems. We propose NEUROLOGICDECODING , an efficient and general method for generating with arbitrary predicate logic constraints. We demonstrate its intuitive application to 4 different tasks as an extension to existing models, showing broad and consistent improvement to decoding quality. 3 We could use any off-the-shelf coreference resolution model for this. However, since the English examples in Stanovsky et al. (2019) follow the Winograd schemas format, we use a RoBERTa model finetuned on Winograd Schema Challenge for this, with 78.4% accuracy. We thank the anonymous reviewers and meta-reviewers for their helpful feedback. This research was supported in part by DARPA under the MCS program through NIWC Pacific (N66001-19-2-4031) and the Allen Institute for AI (AI2).
abstain abstain objective abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain result objective abstain abstain objective objective abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain abstain abstain abstain method abstain method method method method abstain method abstain abstain method abstain abstain abstain abstain abstain method method abstain method other other other other other other other other other other abstain other other other other other other other other other other other other other other other other other other other other other other method other other other other other other other other other method other objective method method method objective other other other other other other other other other other other abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain objective objective method method other other