The ability to sequence unordered events is evidence of comprehension and reasoning about real world tasks/procedures. It is essential for applications such as task planning and multi-source instruction summarization. It often requires thorough understanding of temporal common sense and multimodal information, since these procedures are often conveyed by a combination of texts and images. While humans are capable of reasoning about and sequencing unordered procedural instructions, the extent to which the current machine learning methods possess such capability is still an open question. In this work, we benchmark models' capability of reasoning over and sequencing unordered multimodal instructions by curating datasets from online instructional manuals and collecting comprehensive human annotations. We find current state-of-the-art models not only perform significantly worse than humans but also seem incapable of efficiently utilizing multimodal information. To improve machines' performance on multimodal event sequencing, we propose sequence-aware pretraining techniques exploiting the sequential alignment properties of both texts and images, resulting in >5% improvements on perfect match ratio. Instructions are essential sources for agents to learn how to complete complex tasks composed of multiple steps (e.g., making a wood sign from scratch). However, instructions do not always come in a proper sequential order, for example, when instructions must be combined across sources (e.g., to accomplish a complex task there might be multiple useful resources for certain task-steps come out from a single Google search). Therefore, sequencing unordered task-steps is crucial for comprehending and inferring task procedures, which requires thorough understanding of event causal and temporal common sense. It is essential for Sand the wood block. applications such as multi-source instruction summarization and robot task planning (Garattoni and Birattari, 2018). Existing work has studied sequencing unordered texts from paper abstracts or short stories (Chen et al., 2016; Cui et al., 2018). However, real-life tasks are often complex, and multimodal information is usually provided to supplement textual descriptions to avoid ambiguity or illustrate details that are hard to narrate, as illustrated in Figure 1. To investigate whether current AI techniques can efficiently leverage multimodal information to sequence unordered task instructions, we curate two datasets from online instructional manuals (Hadley et al.; Yagcioglu et al., 2018). We consider two representative instruction domains: cooking recipes and How-To" instructions (WikiHow). We establish human performance for the sequencing task on a subset of each data resource. As certain steps to perform a task can potentially be interchangeable, 1 we collect annotations of possible orders 1 For example, without special requirements, preparing certain ingredients of a dish, such as slicing carrots or cucumbers, does not necessarily need to follow a specific order. 4525 alternative to the originally authored ones to create multiple references . Such additional annotation provides not only better measurement of human and model performance by alleviating unintended biases from content creators, but also a useful resource for future research of models that are aware of task-step dependencies and interchangeability. To measure the ability of state-of-the-art AI techniques to sequence instruction steps, we construct models consisting of: (1) an input encoder which encodes image, text, or multimodal inputs, and (2) an order decoder which predicts step order using the encoded representations. They are jointly trained with the order supervisions. Our preliminary studies show that multimodal information is consistently helpful for the sequencing task. However, compared to humans, current models are less efficient in utilizing multimodal information. We hypothesize that it is because the models do not effectively capture the sequential information in the vision modality nor the sequential alignment between multimodal contents. To address this, we propose to equip models with capabilities of performing sequential aware multimodal grounding . Specifically, we propose several self-supervised objectives, including sequence-based masked language modeling, image region modeling, and content swapped prediction, to pretrain the models before finetuning them on the downstream sequencing task. The proposed pretraining techniques are shown to be effective in improving multimodal performance, enjoying a >5% improvement on the perfect match ratio metric. However, it is still significantly behind human performance (  15 % in perfect match ratio metric). The same trend is observed when alternative orders are considered. Our key contributions are two-fold: (1) We propose a multimodal sequencing task with two cu-rated instructional manuals, and comprehensive human annotations. (2) We investigate model performance on sequencing unordered manuals, and propose sequence-aware pretraining techniques to more effectively use the multimodal information. Our experiments and extensive analysis provide insights on which task categories are most challenging for the state-of-the-art models. They also shed the light that more sophisticated sequential multimodal grounding are required to further improve the performance for the proposed multimodal sequencing task. 2 Problem Definition Given a task procedure S consisting of N steps, where each step S i  S can consist of two types of contents: a textual description T i of tokens { T i,k } n T k =1 and/or image(s) I i = { I i,k } n I k =1 . 2 A model is required to take as inputs a random permutation of S , i . e . S p = { S p 1 , ..., S p N } , where p is a permutation ( S p j can take one of the following three modalities: T p j , I p j , and { T p j , I p j } ), and predict the correct order of S p , i . e . argsort ( S p ) . 3 Datasets and Human Annotation We are interested in understanding the current state-of-the-art models' performance on this multimodal instruction sequencing task. To this end, we curate instruction datasets to support our study. 3.1 Instruction Manual Datasets There are three major features we require for the target datasets: (1) It is multimodal. (2) It consists of task procedures as sequences of steps. (3) Different modalities are used intentionally to complement each other. In light of these, we consider the following two datasets: RecipeQA. We start from a popular as well as intuitive choice of instruction manuals, recipes, which fully fulfill the aforementioned criteria. RecipeQA is a multimodal question answering dataset consisting of recipes scraped from Instructables.com (Yag-cioglu et al., 2018). We utilize the recipes collected in RecipeQA and convert each unique recipe into sequential multimodal steps for our task. WikiHow. To expand the types of instruction manuals for our task beyond recipes, we also consider a popular How To ..." type of instructions, WikiHow, which is an online knowledge base that consists of human-created articles describing procedures to accomplish a desired task. Each article contains a high level goal of a task, a short summary of the task procedures, and several multimodal steps where each step consists of a description paired with one or a few corresponding images. We scrape the entire WikiHow knowledge resource, containing more than 100k unique articles (mostly) with multimodal contents , as well as the hierarchically structured category for each article. Table 1 presents the essential statistics of the two datasets (more details are in Append. Sec. A). To ensure the validity of our proposed multimodal sequencing task, we establish the human performance via Amazon Mechanical Turk. Since our dataset is constructed from resources that are not directly designed for the sequencing task, the quality of random samples is unverified. Specifically, some articles in WikiHow may not have a notion of proper order among the steps. 3 As a result, to construct a high quality test set particularly for WikiHow for establishing human performance, we first identify a set of categories which are more likely to feature proper order, e . g . Home and Garden and Hobbies and Crafts . 4 A random proportion is then sampled and the co-authors further downsam-ple the subset to 300 samples with the aforementioned criteria via majority vote. For RecipeQA, we randomly sample 100 recipes from the dataset. And hence, the resulting two subsets serve as our golden-test-set for performance benchmarking. Human Performance. Prompted with a task goal and a randomly scrambled sequence of the task-steps (can be one of the following modalities: mul-3 No temporal or other dependencies among the task-steps, e . g . How to be a good person, where each step depicts a different aspect and tips of being a good person. 4 Although the data used for training is not cleansed and thus can be noisy, we believe models can still learn to sequence from many of the articles designed to have proper order. timodal or text/image-only), workers are asked to examine the contents and decide the proper performing order. Human performance are then computed against the original authored orders as the ground truths, averaged across the whole set. 5 Alternative Orders. When performing a task, some steps can be interchangeable. To take the interchangeability into consideration in our benchmark task, we also collect possible alternative orders to the original ones to create multiple references. For each instance in our golden-test-set, given the instruction steps sequenced in their original order, we ask workers to annotate alternative orders if the presented task-steps can be performed following a different order. 6 Although in this work we are mainly focusing on sequential instructions and hence the interchangeability is also gauged in a sequential manner, we want to point out that the nature of task-step interchangeability is also highly related to parallel (branching) steps of tasks (Sakaguchi et al., 2021). We argue that the actions that can be performed interchangeably imply no direct dependencies are among these actions and thus can potentially be parallelized, and hence our alternative order formulation can help inferring these parallel actions. More details of the two human annotation tasks can be found in Append. Sec. B. 4 Models To benchmark the proposed task, we construct models comprising: (1) an encoder which encodes multimodal or text/image-only inputs, and (2) an order decoder which utilizes the encoded representations to predict the orders. To help models capture sequentiality in task-steps better as well as adapt to our target task domains, we pretrain the encoders with several self-supervised objectives on the instructions before integrating them with the decoder. Text-Only Encoders. We use RoBERTa (Liu et al., 2019) for text-only inputs. Although the next-sentence prediction in BERT (Devlin et al., 2019) 5 We design an algorithm to compute the inter-annotator agreements (IAAs), see Append. Sec. B.3 for details. The IAAs for ( multimodal , text-only , image-only ) versions in WikiHow is: (0.84, 0.82, 0.69), and (0.92, 0.87, 0.81) in RecipeQA. 6 The alternative order annotation IAAs for ( multimodal , text-only , image-only ) versions in WikiHow is: (0.73, 0.71, 0.78), and (0.79, 0.76, 0.79) in RecipeQA. can potentially be exploited for sequencing, we empirically find that RoBERTa performs better. Multimodal Encoders. We consider the following two V&L models mainly due to their easy adaptation to our proposed sequencing task: VisualBERT (Li et al., 2019) grounds object detected image regions ( e . g . by Faster-RCNN (Ren et al., 2016)) to language with a single transformer model (Vaswani et al., 2017). VisualBERT is pretrained with: (1) multimodal masked language modeling (MLM) 7 , and (2) image-text matching prediction (ITM), where the image in an image-caption pair is randomly replaced with another one to create misalignment, and the model is required to predict whether the current pair is aligned. CLIP-ViL (Shen et al., 2021) is also a single-stream V&L model similar to VisualBERT, while the visual encoder is replaced by a patch-based model inspired by the ViT (Dosovitskiy et al., 2021) in CLIP (Radford et al., 2021), where the image features are taken as gridded-image-patches as shown in Figure 2. The pretraining objectives remain the same as VisualBERT. Empirically, both Shen et al. (2021) and this work find such patch-based model tends to yield better downstream performance. Image-Only Encoders. We attempt to provide an image-only baseline on our sequencing task with two visual encoders: (1) ResNet -based (He et al., 2016) Faster-RCNN model (also the visual encoder in VisualBERT) where both the detected regional features and the whole-image-feature are used, and (2) the aforementioned patch-based CLIP model. 8 7 RoBERTa is used to initialize VisualBERT and CLIP-ViL. 8 Without confusion, throughout the paper we term the ViTand CLIP-inspired visual encoder simply as CLIP. The standard multimodal grounding techniques (Li et al., 2019; Lu et al., 2019; Su et al., 2020; Chen et al., 2020a) do not explicitly concern the sequentiality of text and associated image sequences, and hence may fall short of effectively utilizing the sequential properties in multimodal inputs. To encourage models to have better awareness of the sequential alignments in multimodal instruction steps, we propose to pretrain the encoders with the following self-supervised objectives: (1) masked language modeling ( MLM ), (2) (patch-based) image-swapping predictions ( ISP/PISP ), and (3) sequential masked region modeling ( SMRM ). Figure 2 illustrates an overview of the pretraining paradigm. For the proposed objectives, the inputs to the models are generally ordered instruction step sequences, which can be further sub-sampled to produce length-varying subsequences. Although we do not find this necessarily benefit the downstream performance, it is observed that the sub-sampling helps the model converge faster. While all of our proposed objectives can be applied to sequence with arbitrary length (  2 ), without loss of generality and for simplicity, the following sections assume the sub-sampled sequence is of length 2 . The standard MLM (Devlin et al., 2019) is employed by the text-only models to adapt a pretrained language model to the target domain (task instructions). Following prior V&L works, we apply MLM to multimodal models. Specifically, we ensure that the textual description of each step T i gets similar amount of tokens being masked-out such that the models can potentially exploit the 4528 image sequences more. This objective concerns, with certain probability, randomly swapping a pair of items in a sequence and asking the model to judge whether the resulting sequence is properly ordered or not ( i . e . binary classification). We mainly perform the swapping in the image modality and hence it can be viewed as a sequence-aware version of ITM objective in most V&L models. As in ITM, the output representation at the [CLS] token is used to make the prediction. Standard. For an ordered sequence S , we can randomly swap two 10 items of S , { S i , S j } , where i < j , to { S j , S i } , with a certain probability  . Our preliminary studies find that swapping the textual contents does not necessarily help the downstream performance for either text-only or multimodal models, so we only perform the swapping on the images { I i , I j } in both multimodal and image-only models. For patch-based image inputs (or regional features), the whole patches of an image are swapped with those of another one within the same sequence, as illustrated in Obj 2 in Figure 2. Patch-Based. We can perform the aforementioned swapping prediction with a finer granularity, directly on the image patches. Assuming each image I i is cropped into w patches (or w detected regions), i . e . { i i,k } wk =1 = { i i, 1 , ..., i i,w } , we randomly select M (ranging from 1 to w ) number of patches each from the two images I i , I j ( i . e . { i i,p } , { i i,q } , p, q  M -sized sampled indices) to be swapped with probability  . Specifically, for each image patch i i,m  I i , a randomly selected image patch i j,n  I j is sampled to be swapped with. The sampled M -sized indices do not need to be the same set of integers for each image. The Obj 3 in Figure 2 illustrates the patch-based swapping prediction with w = 4 and M = 2 . Prior works extend the masked learning to the visual modality, where the masked target is either a predefined discrete visual vocabulary (Sun et al., 2019; Bao et al., 2021) or (soft) object class labels (Lu et al., 2019; Su et al., 2020; Chen et al., 2020a). In this work, we construct a feature-based target vocabulary dynamically in each training batch. We first randomly select the same amount 9 As higher chances that the complementary textual information is also masked out from different steps. of X % ( X = 15 ) patches for each image to be masked out (replaced with 0-tensor), and then construct a target vocabulary from the original output representations (before masking) of these patches. Concretely, denote the output representation of an input image-patch i i,m as h ( i ) i,m and the masked positions of I i as D i , we can construct a candidate list from all the output representations of the patches at the masked positions of each image, i . e . C = { h ( i ) i,m }{ h ( i ) j,n } , m, n  D i , D j . Denote the masked image patches (the gray-colored image patches in Figure 2) as mask(i) i,m , for each output masked representation h ( mask(i) ) i,m , we concatenate it with all the candidates, i . e . h ( mask(i) ) i,m || h ( i' ) ,  i'  C , which results in | C | concatenated representations for each masked position. A | C | -way multi-class classification can then be performed by maximizing the probability of p ( i i,m | h ( mask(i) ) i,m ; C ) . For robust training, we additionally: (1) shuffle the candidate set C for each masked position to prevent overfitting, and (2) ensure the overlapping of masked positions in each pair of images, D i  D j , is < 50%, allowing the models to utilize information of similar regions from other images in the sequence. As the mechanism in some objectives cannot guarantee mutually exclusive impacts ( e . g . performing ISP and PISP simultaneously may create confusing swapped patches), we employ a turn-taking fashion, with uniform probability, one of the objectives ( Obj ) is sampled for each training mini-batch. The overall pretraining objective is defined as below: L = LMLM + L Obj , Obj  { ISP , PISP , SMRM } (1) 4.3 Order Decoder  BERSON BERSON is a recently proposed state-of-the-art neural sentence ordering framework (Cui et al., 2020), where a pointer network (Vinyals et al., 2016) exploits both the local (relative pairwise order) and global (self-attentions on top of the entire input sequence) information of the inputs to decode the predicted order. BERSON mainly exploits the [CLS] output representations for relational understanding, which aligns well with how our encoders are pretrained (Figure 2). We integrate our encoders (with or without sequence-aware pretraining) into BERSON, replacing its original BERT encoder. The BERSON-module-specific components are freshly initialized and then the entire integrated module is finetuned on our sequencing task. Our experiments seek to answer these questions: (1) How valid is the proposed task for humans to complete? (2) Is multimodality helpful? (3) Can the proposed sequence-aware pretraining utilize multimodality more effectively? (4) How would results differ when alternative orders are considered? Position-Based metrics concern the correctness of the absolute position of each item in a sequence, including: (1) Accuracy (Acc) which computes the ratio of absolute positions in the ground truth order that are correctly predicted; (2) Perfect Match Ratio (PMR) which measures the percentage of predicted orders exactly matching the ground truth orders; and (3) Distance (Dist.) which measures the average distance 11 between the predicted and ground truth positions for each item. Longest Common Subsequence computes the average longest subsequences in common (Gong et al., 2016) between the predicted and ground truth orders ( L q ). We also consider a stricter version, longest common substring, which requires the consecutiveness for the comparisons ( L r ). Kendall's Tau (  ) (Lapata, 2003) is defined as 1  2  (# inversions ) / (# pairs ) , where the inversion denotes that the predicted relative order of a pair of items is inverted compared to the corresponding ground truth relative order, and # pairs = (cid:0) N 2 (cid:1) for N -length sequence. Each metric focuses on different perspectives of the predictions, i . e . position metrics concern the absolute correctness, while common subsequence and  metrics measure if general sequential tendency is preserved despite incorrect absolute positions. We use the original data splits for RecipeQA. For WikiHow, to prevent models' exploiting knowledge from similar articles, we split the data so that certain (sub)categories do not overlap in each split. We use only the train splits in each dataset to perform their respective pretraining. More details of the data splits are in Append. Sec. A. Preliminary studies show that joint training with both RecipeQA and WikiHow data does not necessarily improve 11 Except for distance metric, higher scores are better. the downstream performance, thus the models evaluated in the two datasets are trained simply using their respective training sets for faster convergence. We cap the overall sequence length at 5 and each step description with maximally 5 sentences for both models and humans. The maximum input length per step is 60 tokens (overall maximum length = 300 ) for training and GPU memory effi-ciency. = 0 . 5 for both ISP and PISP. All images are resized to 224  224 , and 32  32 patch is used for CLIP-based models, resulting in 7  7 = 49 patches per image. Aside from standard positional embedding, we only supplement a modality token type embedding (text := 0, image := 1) to the multimodal models. Pretrained weights for each encoder is obtained either from their corresponding code bases or by running their codes on our setup. 12 5.3 Standard Benchmark Results Table 2 summarizes both the human and model performance for each input modality evaluated using the original ground truth orders on the golden-test-set, whereas Table 3 summarizes a more detailed breakdown of the model performance when incrementing combinations of pretraining objectives. As is shown, multimodal information is veri-fied consistently helpful for humans. Compared under same scenario with or without the sequence-aware pretraining, the two multimodal models consistently outperform their text-only counterparts, where the proposed pretraining technique is shown particularly effective for the patch-based multimodal model (CLIP-ViL). However, our top-performing models still exhibit significant gaps below human performance, especially in PMR. Additionally, we observe a different trend in the two datasets where the multimodality benefits more in RecipeQA than WikiHow. The gap between the multimodal human and model performance is larger than the text-only counterparts in WikiHow, while a reversed trend is shown in RecipeQA. We hypothesize that recipes may contain more domain-specific language usages and/or less words for the pretrained language models and hence benefits more from the our in-domain sequence-aware pretraining. Humans, on the other hand, benefit more from the images in WikiHow as its texts are hypothesized to contain more ambiguities. models perform closer to humans, and on which the multimodal information is most efficiently utilized. In Figure 3 we select categories with the top and least performance gaps (with PMR metric, top=3, least=2) between the human and our best performing models. We observe that the categories on which multimodal models outperform the text-only ones the most are also the categories the models perform closest to humans, e . g . Home and Garden . We hypothesize that the images in these categories are well complementary to the texts and that our sequence-aware grounding performs effectively. In contrast, in categories such as Arts and Entertainment and Hobbies and Crafts where humans still enjoy benefits from multimodal information, our models have difficulty utilizing the multimodal information. We hypothesize that better visual understanding may alleviate the potentially suboptimal grounding as images of these categories can contain many non-common objects. For each instance where alternative ground truth orders exist, the performance is computed by the best each predicted order can obtain against all the ground truth orders 13 , denoted by multi-reference performance , and the subset containing these instances is denoted as the multi-reference subset . Statistics. Table 5 lists the essential statistics of the multi-reference subsets, including the counts of the multi-reference instance for each dataset and modality, as well as the per-instance statistics. Multi-Reference Performance. The noticeable main competitors in Table 2 are multimodal and text-only models, and hence for conciseness, in Table 4 we mainly report the multi-reference version 13 Jointly considered from all the evaluation metrics. 14 The overall average number of ground truth references becomes 1.19, 1.23, 1.09 for multimodal, text-only, and image-only versions in WikiHow; and 1.10, 1.17, 1.14 in RecipeQA. of their best performing variants with the selected metrics. Several trends still hold: (1) Multimodal models still outperform the text-only counterparts. (2) Human performance is still well above models' even under multi-reference setups. Additionally, both humans and models perform significantly worse in the multi-reference subset when single (original) ground truth is enforced, implying the validity of our alternative order annotations. We originally hypothesize that enforcing the original authored order to be the only ground truth would be unfair to the text-only models, as images can often better represent the detailed scene changes omitted by the texts, while in reality certain steps may not need to strictly follow the authored order. Judging from the number of instances that improve after evaluating with alternative orders, the text-only model indeed benefits more from the multi-reference setup. Examining the general trends in Table 4, one can conclude that the textual contents indeed posses certain levels of ambiguities where images can help to alleviate. However, as the performance gaps between multimodal and text-only models are still significant under the multi-reference settings, advantages of multimodality. Note that humans achieve perfect performance on the multi-reference subset in RecipeQA, though unlikely it may seem, it is mainly due to recipes tend to have rarer possible alternative orders. WikiHow Categories. Table 6 lists the WikiHow categories with the most (top-5) annotated multi-reference ground truths. Note that the categories with more annotated alternative ground truths are also among the worse performance from both humans and models (Figure 3). We provide sample qualitative inspections in Append. Sec. C.1. sequential reasoning which is shown evident for procedural understanding (Tomkins, 1952; Baron-Cohen et al., 1986; Loucks et al., 2017). In NLP, existing works attempt the sequencing task as sorting a series of unordered sentences (Chen et al., 2016; Cui et al., 2018; Logeswaran et al., 2018; Oh et al., 2019; Lee et al., 2020; Calizzano et al., 2021) from paper abstracts or short paragraphs. While certain prior work also attempts to extend it to incorporate multimodality (Agrawal et al., 2016), the dataset used, Visual StoryTelling (Huang et al., 2016), features album images that were not intended to be procedural nor supply unstated details to complement the texts. In computer vision, existing work leverages shuffle frame prediction for learning video representations (Lee et al., 2017; Xu et al., 2019; Wang et al., 2020; Li et al., 2020) as well as cycle consistency constraints for learning temporal dynamics (Epstein et al., 2021). Zellers et al. (2021) also features a pairwise relative frame re-ordering objective to learn temporal common sense from scripted videos, however, as their downstream tasks mainly concern visual reasoning and ordering by frame-text-matching (also on Visual StoryTelling), the re-ordering objective is more focused on the visual modality. Our work takes a different perspective to tackle a comprehensive multimodal sequencing task with a focus on the procedural task-solving knowledge and gauging the helpfulness of complementary information in different modalities. Task/Procedure Understanding. Other works have utilized WikiHow for learning task knowledge. In NLP, textual descriptions of WikiHow have been used for abstractive summarization (Koupaee and Wang, 2018), procedural understanding (Zhou et al., 2019; Tandon et al., 2020), and intent estimation (Zhang et al., 2020a). Prior work (Zhang et al., 2020b) considers WikiHow for learning event temporal ordering, but limited to only pairwise relations. A concurrent work uses WikiHow to infer visual goals (Yang et al., 2021). We hope our curation can help advancing the goal of comprehensive multimodal procedural understanding. Another popular form of comprehending given procedures is through a multiple choice machine comprehension task. Prior work has utilized text book figures (Kembhavi et al., 2017) as a holistic "reading reference" for models to select the correct order of certain (textually described) events from given multiple choices . Another work attempts the original visual ordering task of RecipeQA (Liu et al., 2020) (also an multiple choice task). However, we argue that our task tackles a more complex task as the desired orders need to be directly derived and the event-wise complementary multimodal understanding is not an essential component in these existing works. Multimodality. Beside models used in this work, there are several recent advanced multimodal grounding techniques (Tan and Bansal, 2019; Li et al., 2019; Lu et al., 2019; Su et al., 2020; Chen et al., 2020b; Huang et al., 2020; Wen et al., 2021). We utilize VisualBERT and CLIP-ViL for their simplicity to be adapted to our task and easier integration to our proposed pretraining techniques, however, our framework is able to incorporate any of the aforementioned multimodal models. In this work we present studies of language and multimodal models on procedure sequencing, leveraging popular online instructional manuals. Our experiments show that both multimodality and our proposed sequence-aware pretraining are helpful for multimodal sequencing, however, the results also highlight significant gaps below human performance (  15% on PMR). We provide insights as well as resources, such as the multi-reference annotations of the sequencing task, to spur future relevant research. We also anticipate that the alternative orders defined and annotated in our work can benefit more comprehensive task-procedure understanding. Future work such as predicting task steps which can be parallel or interchangeable, and understanding step dependencies can be explored. Many thanks to Liunian Harold Li for his original CLIP-ViL implementation; to I-Hung Hsu and Zi-Yi Dou for their helpful discussion and feedback; and to the anonymous reviewers for their constructive suggestions. This material is based on research supported by the Machine Common Sense (MCS) program under Cooperative Agreement N66001-19-2-4032 with the US Defense Advanced Research Projects Agency (DARPA) and a CISCO research contract. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing DARPA, CISCO, or the U.S. Government. We hereby acknowledge that all of the co-authors of this work are aware of the provided ACM Code of Ethics and honor the code of conduct. This work is mainly about sequencing a given series of multimodal task procedures, represented by text descriptions along with their images. The followings give the aspects of both our ethical considerations and our potential impacts to the community. Dataset. We collect the human performance on our sequencing task (both the standard human performance and the alternative order annotations) via Amazon Mechanical Turk (MTurk) and ensure that all the personal information of the workers involved (e.g., usernames, emails, urls, demographic information, etc.) is discarded in our dataset. While the sequence orders either from the original author intended ones or those annotated by the workers for the standard performance may possess unintended biases against certain population group of people ( e . g . due to cultural differences or educational differences, some tasks may be performed differently from the original intended orders), we anticipate the additional multi-reference annotation can alleviate such an issue as well as provide a broader view to approach procedural understanding, i . e . certain task-steps can be interchanged. This research has been reviewed by the IRB board and granted the status of an IRB exempt . The detailed annotation process (pay per amount of work, guidelines) is included in the appendix; and overall, we ensure our pay per task is above the the annotator's local minimum wage (approximately $12 USD / Hour). We primarily consider English speaking regions for our annotations as the task requires certain level of English proficiency. Techniques. We benchmark the proposed sequencing task with the state-of-the-art large-scale pretrained language and multimodal models with our novel sequence-aware pretraining techniques. As commonsense and task procedure understanding are of our main focus, we do not anticipate production of harmful outputs, especially towards vulnerable populations, after training models on our proposed task.
abstain abstain abstain abstain method result objective abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain method method method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain other abstain method method abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method objective method method abstain other other other method abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain