Many high-level procedural tasks can be decomposed into sequences of instructions that vary in their order and choice of tools. In the cooking domain, the web offers many partially-overlapping text and video recipes (i.e. procedures) that describe how to make the same dish (i.e. high-level task). Aligning instructions for the same dish across different sources can yield descriptive visual explanations that are far richer semantically than conventional textual instructions, providing commonsense insight into how real-world procedures are structured. Learning to align these different instruction sets is challenging because: a) different recipes vary in their order of instructions and use of ingredients; and b) video instructions can be noisy and tend to contain far more information than text instructions. To address these challenges, we first use an unsupervised alignment algorithm that learns pairwise alignments between instructions of different recipes for the same dish. We then use a graph algorithm to derive a joint alignment between multiple text and multiple video recipes for the same dish. We release the MICROSOFTRESEARCHMULTIMODALALIGNEDRECIPECORPUS 1 containing  150K pairwise alignments between recipes across 4,262 dishes with rich commonsense information. Although machine learning has seen tremendous recent success in challenging game environments such as Go (Schrittwieser et al., 2019), DOTA (Ope-nAI, 2019), and StarCraft (DeepMind, 2019), we have not seen similar progress toward algorithms that might one day help humans perform everyday tasks like assembling furniture, applying makeup, repairing an electrical problem, or cooking a particular dish. In part this is because the relevant large-scale multimodal (language, video, audio) datasets are difficult to acquire, even with extensive crowdsourcing (Salvador et al., 2017; Sanabria et al., 2018). Unimodal data, though, is abundant on the web (e.g. instructional videos or textual instructions of tasks). Using language as the link between these modalities, we present an approach for learning large-scale alignment between multimodal procedural data. We hope our work, and the resulting released dataset, will help spur research on real-world procedural tasks. Recipes in the cooking domain provide procedural instruction sets that are captured  in large volume  both in video and text-only forms. Instruction sets in these two modalities overlap suf-ficiently to allow for an alignment that reveals interestingly different information in the linguistic and visual realms. In Figure 1, for instance, the text recipe (left) and the transcribed video recipe (right) for shrimp fried rice vary in word usage, order of instructions and use of ingredients. Know-5. ing that the highlighted instructions correspond to the same step is useful in understanding potential ingredient substitutions, how the same step can be linguistically described and physically realized in different ways, and how instruction order can be varied without affecting the outcome. Motivated by this idea that aligned procedural data can be a powerful source of practical commonsense knowledge, we describe our approach for constructing the MICROSOFTRESEARCHMULTIMODALALIGNEDRECIPECORPUS . We first extract a large number of text and video recipes from the web. Our goal is to find joint alignments between multiple text recipes and multiple video recipes for the same dish (see Figure 2). The task is challenging, as different recipes vary in their order of instructions and use of ingredients. Moreover, video instructions can be noisy, and text and video instructions include different levels of specificity in their descriptions. Most previous alignment approaches (Munteanu and Marcu, 2005) deal with pairwise alignments. Since our goal is to align multiple instruction sets, we introduce a novel two-stage unsupervised algorithm. In the first stage, we learn pairwise alignments between two text recipes, two video recipes, and between a text and a video recipe using an unsupervised alignment algorithm (  3.1). In the second stage, we use the pairwise alignments between all recipes within a dish to construct a graph for each dish and find a maximum spanning tree of this graph to derive joint alignments across multiple recipes (  3.2). We train our unsupervised algorithm on 4,262 dishes consisting of multiple text and video recipes per dish. We release the resulting pairwise and joint alignments between multiple recipes within a dish for all 4,262 dishes, along with commonsense information such as textual and visual paraphrases, and single-step to multi-step breakdown (  5). We evaluate our pairwise alignment algorithm on two datasets: 1,625 text-video recipe pairs across 90 dishes from the YouCook2 dataset (Zhou et al., 2018a), and a small set of 200 human-aligned text-text recipe pairs across 5 dishes from Common Crawl. We compare our algorithm to several textual similarity baselines and perform ablations over our trained model (  4). Finally, we discuss how this data release will help with research at the intersection of language, vision, and robotics (  6). We describe our approach for collecting large-scale text and video recipes; and constructing recipe pairs for training our unsupervised alignment algorithm. We extract text recipes from Common Crawl, 2 one of the largest web sources of text. We heuristically filter the extracted recipes 3 to obtain a total of 48,852 recipes across 4,262 dishes. The number 2 https://commoncrawl.org/ 3 Details in supplementary. For each dish in the text recipes, we use the dish name with recipe' appended, e.g. chocolate chip cookie recipe', as a query on YouTube and extract the top N videos where N is proportional to the number of text recipes for that dish 4 to obtain a total of 77,550 video recipes. We transcribe these videos using the Microsoft Speech-to-Text Cognitive service. 5 Video recipes, unlike text recipes, contain noninstructional (chat) information. For instance, the presenter may give an introduction either of themselves or of the dish at the beginning of the video before diving into the steps of the recipe. Figure 3 contains an example transcript with chat and content information marked. We hypothesize that it is useful to remove such chat information from the transcripts before aligning them to text recipes. We build a supervised chat/content classifier using the YouCook2 dataset (Zhou et al., 2018a), an existing instructional cooking video dataset where parts of video that correspond to instructions are annotated by humans. We assume that these parts correspond to content whereas the rest of the video corresponds to chat. 6 We preprocess the transcriptions of all 77,550 videos using this chat/content classifier 7 to remove all sentences classified as chat. Given N text recipes and M video recipes for a dish, we pair each text recipe with every other text recipe to get O ( N 2 ) text-text recipe pairs. Similarly, we pair each text recipe with every video recipe to get O ( N  M ) text-video recipe pairs, and pair each video recipe with every other video recipe to get O ( M 2 ) video recipe pairs. On closer inspection, we find that some of these pairs describe recipes that are very different from one other, making a reasonable alignment almost impossible. For example, one black bean soup recipe might require the use of a slow cooker, while another describes using a stove. We therefore prune these recipe pairs based on the match of ingredients and length 8 to finally yield a set of 63,528 text-text recipe pairs, 65,432 text-video recipe pairs and 19,988 video-video recipe pairs. We split this into training, validation and test split at the dish level. Table 1 shows the number of dishes and pairs in each split. We first describe our unsupervised pairwise alignment model trained to learn alignments between text-text, text-video, and video-video recipes pairs. We then describe our graph algorithm, which derives joint alignments between multiple text and video recipes given the pairwise alignments. Our alignment algorithm is based on prior work (Naim et al., 2014) that learns to align a sequence of natural language instructions to segments of video recording of the same wet lab protocol. They first identify the nouns in the text sentences and the blobs (i.e. objects) in video segments. Given the blobs from M video segments F = [ f (1) , ..., f ( M ) ] and the nouns from N sentences E = [ e (1) , ..., e ( N ) ] , the task is to learn alignments between video segments and text sentences. They propose a hierarchical generative model which first uses a Hidden Markov Model 8 Details in supplementary. (HMM) (Rabiner, 1989; Vogel et al., 1996) to generate each video segment f ( m ) from one of the text sentences e ( n ) . They then use IBM1 model (Brown et al., 1993) emission probabilities to generate the blobs { f ( m ) 1 , ..., f ( m ) J } in f ( m ) from the nouns { e ( n ) 1 , ..., e ( n ) I } in e ( n ) as follows: P ( f ( m ) | e ( n ) ) = (cid:15) ( I ) J J (cid:89) j =1 J (cid:88) i =1 p ( f ( m ) j | e ( n ) i ) (1) The hidden state in the HMM model corresponds to the alignment between video segment and text sentence, and the state transition probabilities correspond to the jump between adjacent alignments. For computational tractability, a video segment can be aligned to only one sentence (multiple sentences can align to the same video segment) We use this algorithm to learn pairwise alignments between text-text, text-video and video-video recipes. Given two recipes ( source and target ) of the same dish, we define our alignment task as mapping each text instruction (or video transcript sentence) in the source recipe to one or more text instructions (or video transcript sentences) in the target recipe. We make two modifications to the alignment algorithm described above: First, our recipe pairs, unlike the wet lab protocol data, does not follow the same temporal sequence. The alignment algorithm must thus learn to jump within a longer range. We set the window of jump probabilities at [  2 , 2] . 9 Second, we use transcriptions to learn alignments rather than the objects detected in videos. We hypothesize that the richness of language used in instructional videos may facilitate better alignment with transcripts (as others have observed (Malmaud et al., 2015; Sener et al., 2015)). We use all words (except stop words) in video transcript sentences and all words in text instructions while learning the IBM1 word level probabilities. An instruction in one recipe can be aligned to multiple instructions in the other recipe. We use the pairwise alignments to derive a joint alignment at the dish level between multiple text and video recipes. For each dish, we construct a graph where each node represents an instruction from a text recipe or a transcript sentence from a video recipe. We use the pairwise alignments to draw edges between nodes, with alignment probabilities as the edge weights. We include only those edges that have alignment probability greater than 0.5. The pairwise alignments are directed since they go from the source recipe to the target recipe. We first convert the directed graph into an undirected graph by averaging the edge weights between two nodes and converting directed edges into undirected edges. Note that the resultant graph can have multiple connected components as some recipe pairs may not have any instructions aligned with probability greater than the threshold of 0.5 Our goal is to find a set of jointly-alignable instructions across different recipes. We therefore convert the graph (with cycles) into a forest by running the maximum spanning tree algorithm on the graph. Figure 4 shows an example tree derived for one of the dishes. A path in this tree, that has at most one node from each recipe, constitutes a set of jointly-alignable instructions. For example, in the magnified section of the tree in Figure 4, all unique colored nodes in the path from the yellow node to the green node constitute a set of jointly-alignable instructions. We describe how we evaluate our pairwise alignment algorithm (from  3.1). We answer the following research questions using our experimentation: 1. How does our alignment model perform when evaluated on human-aligned recipe pairs? 2. Does our unsupervised alignment model outperform simpler non-learning baselines? 3. How does performance differ when we use only nouns or nouns and verbs instead of all words to learn alignments? YouCook2 text-video recipe pairs The YouCook2 dataset (Zhou et al., 2018a) consists of 1,625 cooking videos paired with human-written descriptions for each video segment. These span 90 different dishes. We transcribe all videos using the Microsoft Speech-to-Text Cognitive service 10 and separate it into sentences using a sentence tokenizer. Given a sequence of human-written descriptions and a sequence of transcript sentences, the alignment task is to align each transcript sentence to one of the human-written descriptions. We train our pairwise alignment model on the train split of our text-video recipe pairs (from 10 https://azure.microsoft.com/ en-us/services/cognitive-services/speech-to-text/  2.3) and evaluate on the YouCook2 dataset. An important difference between the text-video pairs in YouCook2 and in our data is that in YouCook2, the text instructions and the video segments are temporally aligned since the text instructions were specifically written for the videos. In our data, however, the text and the video recipes can differ in order. CommonCrawl text-text recipe pairs We randomly choose 200 text-text recipes pairs (spanning 5 dishes) from the test split of our data (  2.3) and collect alignment annotations for them using six human experts. We show annotators a numbered list of the instructions for the target recipe (along with its title and ingredients). We display instructions for the source recipe with input boxes besides them and ask annotators to write in the number(s) (i.e labels) of one or more target instruction(s) with which it most closely aligns. Each recipe pair is annotated by three annotators. For 65% of the instructions, two or more annotators agree on a label. For only 42% of the instructions do all three annotators agree, suggesting that the difficulty level of this annotation task is high. We train our pairwise alignment model on the train split of our text-text recipe pairs (  2.3) and evaluate on the 200 human-aligned pairs. Baselines described below align each instruction 11 in the source recipe to one or more instructions in the target recipe. Random We align each instruction in the source recipe to a random instruction in the target recipe. Uniform alignment Given N instructions in the target recipe, we divide the instructions in the source recipe into N equal chunks and align each instruction in the i th chunk of the source recipe to the i th instruction in the target recipe. For instance, given a source recipe [ S 1 , S 2 , S 3 , S 4] and a target recipe [ T 1 , T 2] , uniform alignment would align S 1 and S 2 to T 1 and S 3 and S 4 to T 2 . More generally, we align the i th instruction in the source recipe to the [( NM i ) th  ( NM ( i + 1)) th ) instruction in the target recipe. BM25 retrieval We use BM25 (Robertson et al., 2009) as our information retrieval baseline. Given 11 We use the term instruction to mean both text instruction and transcript sentence. a source and a target recipe pair, we construct a corpus using all instructions in the target recipe. We then use each source instruction as a query to retrieve the top most instruction from the target instruction corpus and align the source instruction to the retrieved target instruction. Textual similarity Given a source recipe instruction and a target recipe instruction, we define a measure of textual similarity between the two instructions using the following five methods. For each source instruction, we compute its similarity score with every target instruction and align it to the target instruction with the highest score. a. Exact word match : Given two instructions, we define exact word match as the ratio of the number of common words between the two divided by the number of words in the longer of the two. This gives us a measure of word match that is comparable across instructions of different lengths. b. TF-IDF : We use all the recipes in our training set to create a term frequency (TF)-inverse document frequency (IDF) vectorizer. Given an instruction from the evaluation set, we compute the TF-IDF vector for the instruction using this vectorizer. Given two instructions, we define their TF-IDF similarity as the cosine similarity between their TF-IDF vectors. c. GloVe : We train GloVe embeddings (Pennington et al., 2014) on an in-domain corpus of 3 million words put together by combining text recipes and video transcriptions. Given an instruction, we average the GloVe embeddings (Pennington Methods Precision Recall F1 Random 14.26 14.00 12.69 Uniform alignment 41.38 31.85 33.22 BM25 retrieval 50.06 55.27 49.30 Textual Similarity Exact word match 53.90 48.39 46.98 TF-IDF 52.78 46.82 45.12 GloVe 56.04 51.89 50.30 BERT 50.72 55.07 49.10 RoBERTa 52.49 55.86 50.44 HMM+IBM1Nouns 62.11 48.99 50.73 Nouns+Verbs 64.72 50.76 52.97 All words 66.21 52.42 54.55 Table 3: Results for text-text recipe alignment on Common Crawl dataset. et al., 2014) of nouns and verbs 12 to obtain its embedding vector. Given two instructions, we define their embedding similarity as the cosine similarity of their embedding vectors. d. BERT : Given an instruction, we compute its embedding vector using BERT-based sentence embedding (Reimers and Gurevych, 2019). We experiment with different variants and find that the BERT-base model trained on AllNLI, then on STS benchmark training set 13 performed the best for us. Given two instructions, we define their BERT similarity as the cosine similarity between their sentence embedding vectors. e. RoBERTa : We also experiment with a variant of the above baseline where we use RoBERTa (Liu et al., 2019) instead of BERT to compute the sentence embeddings. We use RoBERTa-large trained on AllNLI, then on STS benchmark training set. HMM+IBM1 (nouns) We use the NLTK 14 part-of-speech tagger to identify all the nouns in an instruction and only use those to learn the IBM1 word-level alignments. This ablation is similar to the model proposed by Naim et al. (2014) that align objects in videos to nouns in text. 12 We find that using only nouns and verbs outperforms using all words. HMM+IBM1 (nouns and verbs) We use both nouns and verbs to learn IBM1 word-level alignments. This ablation is similar to the method used in Song et al. (2016) that align objects and actions in videos to nouns and verbs in text. Given M source recipe instructions and N target recipe instructions, the alignment task is to label each of the M source instructions with a label from [0 , ..., ( N  1)] . Given a predicted sequence of labels (from baseline or proposed model) and a reference sequence of labels (from human annotations) for a recipe pair, we calculate the weighted-average 16 precision, recall and F1 score. We average these scores across all alignment pairs to compute aggregate scores on the test set. On text-video alignments Table 2 shows results of our pairwise alignment algorithm compared with baselines on 1,625 human aligned text-video recipe pairs from YouCook2. The BM25 baseline outperforms two of the textual similarity baselines. Within the textual similarity baselines, RoBERTa outperforms all others suggesting that a pretrained sentence level embedding acts as a good textual similarity method for this alignment task. The uniform alignment baseline, interestingly, outperforms all other baselines. This is mainly because in the YouCook2 dataset, the text instructions and the transcript sentences follow the same order, making uniform alignment a strong baseline. Our unsupervised HMM+IBM1 alignment model significantly outperforms (with p < 0 . 001 ) all baselines. Specifically, it gets much higher precision scores compared to all baselines. Under ablations of the HMM+IBM1 model, using all words to learn alignments works best. On text-text alignments Table 3 shows results of our pairwise alignment algorithm compared with baselines on 200 human-aligned text-text recipe pairs from Common Crawl. Unlike text-video alignments, we find that the uniform alignment 15 Experimental details of HMM+IBM1 model is in supplementary. baseline does not outperform textual similarity baselines, suggesting that the different re-orderings between text-text recipe pairs makes alignment more challenging. Within textual similarity baselines, similar to text-video alignment, RoBERTa outperforms all others. We believe this is because text recipes tend to share similar vocabulary, making it easier to find similar words between two textual instructions. Video narrators tend to use more colloquial language than the authors of text recipes, making it more difficult to learn alignments using word similarities. Interestingly, both BM25 and RoBERTa get higher recall than our best HMM+IBM1 model but they lose out on precision. This suggests that retrieval models are good for identifying more alignments, albeit with lower precision. Our unsupervised HMM+IBM1 model again significantly outperforms ( p < 0 . 001 ) all baselines on F1 score. Under ablations of the HMM+IBM1 model, we again find that using all words to learn alignments performs best. Comparing text-video and text-text alignment results On comparing Table 2 and Table 3, we find that textual similarity baselines have overall higher scores on the text-text alignments than the text-video alignments. Our HMM+IBM1 model, on the other hand, has overall higher scores on text-video alignments than on text-text alignments. We attribute this contrast to the fact that two text recipes have higher vocabulary similarities than a text and a video recipe, resulting in textual similarity baselines to perform well on text-text alignments. Our HMM+IBM1 unsupervised learning model is able to do better on text-video pairs where the word usage differences are higher. Furthermore, the text-video pairs from YouCook2 are temporally aligned whereas the text-text pairs from Common Crawl have several re-orderings making the text-text evaluation set comparatively harder. The supplementary material includes an analysis of alignment outputs. We describe the data released in our MICROSOFTRESEARCHMULTIMODALALIGNEDRECIPECORPUS . In all our released data, for text recipes, we include the actual text of the instructions. Whereas, for video recipes, we release the URL to the YouTube video with timestamps corresponding to the aligned video segments. We release the pairwise alignments between recipes of the same dish (derived from  3.1) for 4,262 dishes. This includes 63,528 alignments between text recipes, 65,432 alignments between text and video recipes; and 19,988 alignments between video recipes. We also release the joint alignments between multiple text and multiple video recipes within a dish (derived from  3.2) for 4,262 dishes. The pairwise alignment algorithm described in  3.1 gives alignment probabilities for each pair of instructions it aligns. We threshold on these alignment probabilities to retrieve textual and visual paraphrases. Since our goal is to extract large number of high quality paraphrases, we decide on the threshold value by looking at the trade-off between the percentage of paraphrases extracted and their quality as measured by human annotators on 200 human-aligned text-text recipe pairs from our evaluation set (  4.1). Figure 5 shows the trade-off between the precision, recall and F1 score and the percentage of paraphrases extracted with increasing threshold on instruction-level alignment probability. At 0 . 5 threshold, we extract 60% of the total alignments as paraphrases from our evaluation set. We use this threshold value of 0 . 5 on the pairwise alignments in the training, validation and test sets to extract a total of 358,516 textual paraphrases and 211,703 text-to-video paraphrases from 4,262 dishes and include it in our corpus. The pairwise alignments between text recipes include many instances where one instruction in one recipe is aligned to multiple instructions in another recipe with high alignment probability (greater than 0 . 9 ). Table 4 shows three such single-step to multistep breakdown. We extract a total of 5,592 such instances from 1,662 dishes across the training, validation and test sets and include it in our corpus. We believe that our data release will help advance research at the intersection of language, vision and robotics. The pairwise alignment between recipes within a dish could be useful in training models that learn to rewrite recipes given ingredient or cooking method based constraints. The joint alignment over multiple text recipes within a dish should prove useful for learning the types of ingredient substitutions and instruction reordering that come naturally to expert cooks. The textual and visual paraphrases will, we believe, have implications for tasks like textual similarity, image and video captioning, dense video captioning and action recognition. The single-step to multi-step breakdown derived from our pairwise alignments may also prove useful for understanding task simplification, an important problem for agents performing complex actions. Such multimodal data at scale is a crucial ingredient for robots to learn-from-demonstrations of procedural tasks in a variety of environments. Collecting such large scale data is prohibitively expensive in robotics since it requires extensive instrumentation of many different environments. Other example applications are learning to ground natural language to physical objects in the environment, and catching when humans are about to commit critical errors in a complicated task and offering to help with corrective instructions. Alignment Algorithms Our unsupervised alignment algorithm is based on Naim et al. (2014), who propose a hierarchical alignment model using nouns and objects to align text instructions to videos. Song et al. (2016) further build on this work to make use of action codewords and verbs. Bo-janowski et al. (2015) view the alignment task as a temporal assignment problem and solve it using an efficient conditional gradient algorithm. Malmaud et al. (2015) use an HMM-based method to align recipe instructions to cooking video transcriptions that follow the same order. Our work contrasts with these works in two ways: we learn alignments between instructions that do not necessarily follow the same order; and our algorithm is trained on a much larger scale dataset. Multi-modal Instructional Datasets Marin et al. (2019) introduce a corpus of 1 million cooking recipes paired with 13 million food images for the task of retrieving a recipe given an image. YouCook2 dataset (Zhou et al., 2018a) consists of 2,000 recipe videos with human written descriptions for each video segment. The How2 dataset (Sanabria et al., 2018) consists of 79,114 instructional videos with English subtitles and crowdsourced Portuguese translations. The COIN dataset (Tang et al., 2019) consists of 11,827 videos of 180 tasks in 12 daily life domains. YouMakeup (Wang et al., 2019) consists of 2,800 YouTube videos, annotated with natural language descriptions for instructional steps, grounded in temporal video range and spatial facial areas. Leveraging Document Level Alignments Our work relies on the assumption that text recipes and instructional cooking videos of the same dish are comparable. This idea has been used to extract parallel sentences from comparable corpora to increase the number of training examples for machine translation (Munteanu and Marcu, 2005; Abdul-Rauf and Schwenk, 2009; Smith et al., 2010; Gregoire and Langlais, 2018). Likewise, Talk-Summ (Lev et al., 2019) use the transcripts of scientific conference talks to automatically extract summaries. Zhu et al. (2015) use books and movie adaptations of the books to extract descriptive explanations of movie scenes. Related Tasks A related task is localizing and classifying steps in instructional videos (Alayrac et al., 2016; Zhukov et al., 2019) where they detect when an action is performed in the video whereas we focus on describing actions. Dense event captioning of instructional videos (Zhou et al., 2018b; Li et al., 2018; Hessel et al., 2019) relies on human curated, densely labeled datasets whereas we extract descriptions of videos automatically through our alignments. We introduce a novel two-stage unsupervised algorithm for aligning multiple text and multiple video recipes. We use an existing algorithm to first learn pairwise alignments and then use a graph-based algorithm to derive the joint alignments across multiple recipes describing the same dish. We release a large-scale dataset constructed using this algorithm consisting of joint alignments between multiple text and video recipes along with useful commonsense information such as textual and visual paraphrases; and single-step to multi-step breakdown. Although our dataset focuses on the cooking domain, our framework should generalize to any domain with abundant volumes of unstructured-but-alignable multi-modal data. DIY (Do-It-Yourself) videos and websites, for instance, are an obvious next target. We also envision extending this work by including audio and video features to enhance the quality of our alignment algorithm. Ultimately, we believe this work will further the goal of building agents that can work with human collaborators to carry out complex tasks in the real world. We would like to thank Harpreet Sawhney, Roshan Rao, Prasoon Goyal, Dilip Arumugam and Raymond J. Mooney for all their help. We would also like to thank the four anonymous reviewers for their useful comments and suggestions.
abstain abstain abstain abstain abstain abstain objective method abstain result abstain abstain abstain method result abstain abstain abstain abstain abstain objective objective objective abstain abstain abstain objective objective result method result method method abstain method method abstain other method abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other method method abstain abstain abstain method abstain method abstain method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain result abstain abstain abstain method abstain other method method method method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result result abstain abstain result abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain method method abstain abstain method method abstain abstain abstain abstain method other other objective other other other other other abstain other other other method method objective objective method method abstain objective objective other other