Procedures are inherently hierarchical. To make videos , one may need to purchase a camera , which in turn may require one to set a budget . While such hierarchical knowledge is critical for reasoning about complex procedures, most existing work has treated procedures as shallow structures without modeling the parent-child relation. In this work, we attempt to construct an open-domain hierarchical knowledge-base (KB) of procedures based on wikiHow, a website containing more than 110 k instructional articles, each documenting the steps to carry out a complex procedure. To this end, we develop a simple and efficient method that links steps ( e.g., purchase a camera ) in an article to other articles with similar goals ( e.g., how to choose a camera ), recursively constructing the KB. Our method significantly outperforms several strong baselines according to automatic evaluation, human judgment, and application to downstream tasks such as instructional video retrieval. 1 1 Introduction A procedure includes some steps needed to achieve a particular goal (Momouchi, 1980). Procedures are inherently hierarchical: a high-level procedure is composed of many lower-level procedures. For example, a procedure with the goal make videos consists of steps like purchase a camera , set up lighting , edit the video , and so on, where each step itself is a procedure as well. Such hierarchical relations between procedures are recursive: the lower-level procedures can be further decomposed into even more fine-grained steps: one may need to arrange the footage in order to edit the video . Relatively little attention has been paid to hierarchical relations in complex procedures in the field  Equal contribution. 1 A demo with partial data can be found at https://wikihow-hierarchy.github.io/. of NLP. Some work performed a shallow one-level decomposition and often required costly resources such as human expert task-specific annotation (Chu et al., 2017; Zhang et al., 2020a, 2021). More attention has been paid in fields adjacent to NLP. For example, Lagos et al. (2017) and Pareti et al. (2014) both create hierarchical structures in how-to documents by linking action phrases in one procedure to another procedure or by linking steps in how-to articles to resources like DBPedia (Auer et al., 2007). This kind of linking is helpful for explaining complex steps to readers who do not have prior knowledge of the topic being explained. In this paper, we revisit this important but understudied task to develop a simple and effective algorithm (Figure 1) to construct a hierarchical knowledge-base (KB) for over 110 k complex procedures spanning a wide range of topics from wikiHow, a large-scale how-to website that has recently become a widely-used resource in NLP (Zhou et al., 2019; Zellers et al., 2019; Zhang et al., 2020d,c). 2 From each wikiHow article which represents a procedure, we follow Zhang et al. (2020d) and extract the title as the goal ( e.g., g 1 in Figure 1), and the paragraph headlines as steps ( e.g., s 1 . . . s n ). Next, we decompose the steps by linking them to articles with the same or a similar goal ( e.g., s 1 to g 2 ). The steps of the linked article are treated as the finer-grained steps ( s i to s j ) of the linked step (s1). In this way, the procedural hierarchies go from shallow (B1) to deep (B4). To link steps and article goals, we employ a retrieve-then-rerank approach, a well-established paradigm in related tasks (Wu et al., 2019; Humeau et al., 2019). Our hierarchy discovery model (3) first independently encodes each step and goal in wikiHow and searches the k nearest goals of similar meaning for each step (B2). Then, it applies a dedicated joint encoder to calculate the similarity score between the step and each candidate goal, 2 www.wikihow.com 2998 : Make videos g 1 : Purchase a camera s 1 : Set up equipment s 2 :Choose a camera g 2 : Consider use case s j : Set a budget s i  B1: Input : Record the video s 3 : Practice editing your videos s 4 cat ( s 1 , g 2 ) cat ( s 1 , g i ) cat ( s 1 , g j ) sim ( s 1 , g i ) = 0.3 sim ( s 1 , g 2 ) = 0.6 sim ( s 1 , g j ) = 0.1  c  b 3.1 g 3 g 4 g k g j g 2 g i s 1 B5: Application 1 (4&5) Enrich wikiHow step-goal hyperlinks S: step collection Purchase a camera Set up equipment Consider use case G: goal collection Make videos Choose a camera Edit videos B2: Candidate retrieval (3.1) B3: Reranking (3.2) Make videos Purchase a camera Set up equipment Consider use case Set a budget g n s j   s i B4: Output g s g g s s  The suggested link is helpful  B6: Application 2 (6) Video retrieval Stain cabinet s (retrieved) !"     Figure 1: The overview of our proposed method. thus reranking the goals (B3). This pipeline can efficiently search over a large candidate pool while accurately measuring the similarity between steps and goals. With each step linked to an article goal, a hierarchical KB of procedures is thus constructed. We evaluate our KB both intrinsically and extrinsically. Intrinsically, the discovered links can be directly used to complete missing step-goal hyperlinks in wikiHow, which have been manually curated (B5). Our proposed method outperforms strong baselines ( e.g., Lagos et al. (2017)) according to both automatic and human evaluation, in terms of recall and usefulness respectively (4, 5). Extrinsically, we consider the task of retrieving instructional videos given textual queries. We observe that queries that encode deeper hierarchies are better than those that do not (6). This provides evidence that our KB can bridge the high-level instructions and the low-level executions of procedures, which is important for applications such as robotic planning. We represent a procedure as a tree where the root node n represents a goal and its children nodes Ch ( n ) represent the steps of n . We formulate the hierarchy discovery task as identifying the steps among Ch ( n ) that can themselves be a goal of some other finer-grained steps (sub-steps), which are inserted into the tree. Figure 1, each article comprises a goal ( g ), and a series of steps ( Ch ( g ) ). Therefore, each article forms a procedure tree of depth one. We denote the collection of all goals and steps in wikiHow as G and S respectively. Our hierarchy discovery algorithm aims to link a step s i  S to a goal g  G such that g has the same meaning as s i . It then treats Ch ( g ) as Ch ( s i ) . Given that g and s i are both represented by textual descriptions, the discovery process can be framed as a paraphrase detection task . This discovery process can be applied recursively on the leaf nodes until the resulting leaf nodes reach the desired granularity, effectively growing a hierarchical procedure tree (B4 of Figure 1). For each of the 1.5 million steps in the wikiHow corpus, we aim to select one goal that expresses the same procedure as the step from over 110 k goals. We propose a simple and efficient method to deal with such a large search space through a two-stage process. First, we perform retrieval , encoding each step and goal separately in an unsupervised fashion and select the k most similar goals for each step s . This process is fast at the expense of accuracy. Second, we perform reranking , jointly encoding a step with each of its candidate goals in a supervised fashion to allow for more expressive contextualized embeddings. This process is more accurate at the expense of speed, since calculating each similarity score requires a forward pass in the neural network. The goal with the highest similarity score is se-2999 lected and the step is expanded accordingly, as in B4 of Figure In the first stage, we independently encode each step s  S and goal g  G with a model M b , resulting in embeddings e s 1 , e s 2 , ..., e s n and e g 1 , e g 2 , ..., e g m . The similarity score between s and g is calculated as the cosine similarity between e s and e g . We denote this first-stage similarity score as sim 1 ( s, g ) . Using this score, we can obtain the topk most similar candidate goals for each step s , and we denote this candidate goal list as C ( s ) = [ g 1 , ..., g k ] . To perform this topk search, we use efficient similarity search libraries such as FAISS (Johnson et al., 2017). We instantiate M b with two learning-based paraphrase encoding models. The first is the SP model (Wieting et al., 2019, 2021), which encodes a sentence as the average of the sub-word unit embeddings generated by SentencePiece (Kudo and Richardson, 2018). The second is SBERT (Reimers and Gurevych, 2019), which encodes a pair of sentences with a siamese BERT model that is finetuned on paraphrase corpus. For comparison, we additionally experiment with search engines as M b , specifically Elasticsearch with the standard BM25 weighting metric (Robertson and Zaragoza, 2009). We index each article with its title only or with its full article. We also experiment with Bing Search API where we limit the search to wikiHow website only 3 . The BM25 with the former setting resembles the method proposed by Lagos et al. (2017). While efficient, encoding steps and goals independently is likely sub-optimal as information in the steps cannot be used to encode the goals and vice-versa. Therefore, we concatenate a step with each of its topk candidate goals in C ( s ) and feed them to a model M c that jointly encodes each step-goal pair. Concretely, we follow the formulation of Wu et al. (2019) to construct the input of each step-goal pair as: [CLS] ctx [ST] step [ED] goal [SEP] where [ST] and [ED] are two reserved tokens in the vocabulary of a pretrained model, which mark the location of the step of interest. ctx is the context for a step ( e.g., its surrounding steps or its goal) that could provide additional information. 3 www.bing.com The hidden state of the [CLS] token is taken as the final contextualized embedding. The second-stage similarity score is calculated as follows: sim 2 ( s, g i ) = proj ( M c ( s, g i )) +  sim 1 ( s, g i ) (1) where proj (  ) takes an d -dimension vector and turns it to a scalar with weight matrix W  R d  1 , and  is the weight for the first-stage similarity score. Both W and  are optimized through back-propagation (see more about labeled data in 4.1). With labeled data, we finetune M c to minimize the negative log-likelihood of the correct goal among the topk candidate goal list, where the log-likelihood is calculated as: ll ( s, g i ) =  log (cid:32) softmax (cid:32) sim 2 ( s, g i ) (cid:80) g j  C ( s ) sim 2 ( s, g j ) (cid:33)(cid:33) (2) Compared to the randomly sampled in-batch negative examples, the topk candidate goals are presumably harder negative examples (Karpukhin et al., 2020) and thus the model must work harder to distinguish between them. We will explain the extraction of the labeled step-goal pairs used to train this model in 4.1. Concretely, we experiment with two pretrained models as M c , specifically BERT-base (Devlin et al., 2019) and DEBERTA -large finetuned on the MNLI dataset (He et al., 2021). We pick them due to their high performance on various tasks (Zhang et al., 2020e). 4 In addition, we consider including different ctx in the reranking input. For each step, we experiment with including no context, the goal of the step, and the surrounding steps of the step within a window-size n ( n =1). Some steps in wikiHow could not be matched with any goal. Such steps are unlinkable because of several reasons. First, the step itself might be so fine-grained that further instructions are unnecessary (e.g. Go to a store ). Second, although wikiHow spans a wide range of complex procedures, it is far from comprehensive. Some goals simply do not exist in wikiHow. Hence, we design a mechanism to predict whether a step is linkable or not explicitly. More specifically, we add a special token unlinkable , 4 https://cutt.ly/oTx5gMM. taken from the reserved vocabulary of a pretrained model, as a placeholder goal to the topk candidate goal list C ( s ) , and this placeholder is treated as the gold-standard answer if the step is determined to be unlinkable. The similarity score between a step and this placeholder goal follows Equation 1 and sim 1 ( s, unlinkable ) is set to the lowest first-stage similarity score among the candidate goals retrieved by the first-stage model. Accurately labeling a step as unlinkable is nontrivial  it requires examining whether the step can be linked to any goal in G . Instead, we train the model to perform this classification by assigning unlinkable to steps that have a ground-truth goal but this goal does not appear in the topk candidate goal list. The loss follows Equation 2. 4 Automatic Step Prediction Evaluation To train our models and evaluate how well our hierarchy discovery model can link steps to goals, we leverage existing annotated step-goal links. In wikiHow, there are around 21 k steps that already have a hyperlink redirecting it to another wikiHow article, populated by editors. We treat the title of the linked article as the ground-truth goal for the step. For example, as in B5 of Figure 1, the ground-truth goal of the step Create a channel is Make a Youtube Channel . We build the training, development and test set with a 7:2:1 ratio. Candidate Retrieval The SP model achieves the best recall of all models, outperforming SBERT by a significant margin. Models based on search engines with various configurations, including the commercial Bing Search, are less effective. In addition, BM25 (goal only), which does not consider any article content, notably outperforms BM25 (ar-ticle) and Bing Search, implying that the full articles may contain undesirable noise that hurts the search performance. This interesting observation suggests that while commercial search engines are powerful, they may not be the best option for specific document retrieval tasks such as ours. The recall@ 30 of the SP model is 72.5%, which bounds the performance of any reranker. 6 As seen in the bottom half of Table 1, reranking is highly effective, as the best configuration brings a 19.6% improvement on recall@ 1 , and the recall@ 10 almost reaches the upper bound of this stage. We find that under the same configuration, DEBERTA -large finetuned on MNLI (He et al., 2021) outperforms BERT by 1.7% on recall@ 1 , matching the reported trends from BERTScore. 5 To qualitatively understand the benefit of the reranker, we further inspect randomly sampled predictions of SP and DEBERTA . We find that the reranker largely resolves partial matching problems observed in SP. As shown in C1 of Table 2, SP tends to only consider the action ( e.g., learn) or the object ( e.g., bike) and mistakenly rank those partially matched goals the highest. In contrast, the reranker makes fewer mistakes. In addition, we observed that the reranker performed better on rare words or expressions. For example, as shown in the last column of C1 , the reranker predicts that vinyl records is closely related to LP records and outputs the correct goal while SP could not. Second, we observe that the surrounding context and the goal of the query step are helpful in general. Incorporating both contexts brings a 3% improvement in recall@ 1 . While steps are informative, 6 We only experiment with SP because it is the best retrieval model, providing a larger improvement headroom. they could be highly dependent on the contexts. For example, some steps are under-specified, using pronouns to refer to previously occurring contents or simply omitting them. The additional information introduced by the context helps resolve these uncertainties. In the first example of C2 , the context minecraft is absent in the query step but present in the goal of that step. Similarly, in the second example, the context eyebrows is absent in the query step but present in both the goal and the surrounding steps. Finally, adding unlinkable prediction harms the recall@ 1 due to its over-prediction of unlinkable for steps whose ground-truth goal exists in the topk candidate list. We also experiment with setting a threshold tuned on the development set to decide which steps are unlinkable, in which case the recall@ 1 degrades from 55.4% to 41.9%. Therefore, this explicit learnable prediction yields more balance between the trade-offs. In 5, we will demonstrate that this explicit unlinkable prediction is overall informative to distinguish steps of the two types through crowdsourcing annotations. We empirically find that setting the weight of sim 1 ( s, g ) (  ) to 0 is beneficial in the unlinkable prediction setting. The automatic evaluation strongly indicates the effectiveness of our proposed hierarchy discovery model. However, it is not comprehensive because the annotated hyperlinks are not exhaustive. We complement our evaluation with crowdsourced human judgments via Amazon Mechanical Turk (MTurk). Each example of annotating is a tuple of a step, its original goal from wikiHow, and the top-ranked e x a c t h e l p f u l r e l a t e d unh e l p f u l 0 50 100 150 200 250 300 350 400 linkable e x a c t h e l p f u l r e l a t e d unh e l p f u l unlinkable DeBERTa-UL DeBERTa SP Figure 2: Crowd workers' ratings of step-goal links predicted by our models. goal predicted by one of our models. For each example, we ask three MTurk workers to judge whether the steps in the article of the linked goal are exact, helpful, related, or unhelpful with regard to accomplishing the queried step. Details about the task design, task requirements, worker pay, example sampling, etc. are in A. We select SP, DEBERTA , and DEBERTA with unlinkable prediction and  = 0 (DEBERTA-UL ) for comparison. We attempt to answer the following questions. First, does the performance trend shown in automatic evaluation hold in human evaluation? Second, can the unlinkable predictions help avoid providing users with misleading information (Rajpurkar et al., 2018)? For the purpose of the second question, we separate the examples into two groups. One contains linkable examples. Namely, those whose top1 prediction is not predicted as unlinkable by the DEBERTA-UL model. Ideally, the linked articles from these examples should be helpful. The other 3002 group contains unlinkable examples. For these, we evaluate the second-highest ranked prediction of the DEBERTA-UL model. Ideally, the linked articles from these examples should be unhelpful. The corresponding crowd judgment is shown in Figure 2. Comparing the models, the DEBERTA model and the DEBERTA-UL model have similar performance, while greatly outperforming the SP model. This shows that our proposed model decomposes much more helpful finer-grained steps to assist users with tasks, similar to the trend observed in our automatic evaluation. Comparing the two graphs, it is apparent that when the DEBERTAUL model predicts unlinkable for a step, the suggested decompositions of all models are more likely to be unhelpful. This implies the high precision of the unlinkable prediction, effectively avoiding misleading predictions. Note that our study does not explicitly require subjects to carry out the task, but only annotates whether they find the instructions helpful. In addition to intrinsic evaluation, we take a further step to study the usefulness of our open-domain hierarchical KB to downstream tasks. We select video retrieval as the extrinsic evaluation task, which aims at retrieving relevant how-to videos for a textual goal to visually aid users. More formally, given a textual goal g , the task is to retrieve its relevant videos v g from the set of all videos, with a textual query q . Intuitively, our KB can be useful because videos usually contain finer-grained steps and verbal descriptions to accomplish a task. Therefore, the extra information presented in decomposed steps could benefit retrieving relevant videos. We use Howto100M (Miech et al., 2019) for evaluation. It is a dataset of millions of instructional videos corresponding to over 23 k goals. We construct our video retrieval corpus by randomly sampling 1 , 000 goals ( e.g., record a video ) with their relevant videos. The relevant videos v g = { v 1 , v 2 , ..., v n } of each goal g in the dataset are obtained by selecting the top 150 videos among the search results of the goal on YouTube. 7 For 7 Although the relevance between a goal and a video is not explicitly annotated in the Howto100M dataset, we argue that with the sophisticated engineering of the YouTube video search API and hundreds of thousands user clicks, the highly Query R/P@1 R/P@10 R/P@25 R/P@50 MRL 0 2.2/89.2 19.2/78.1 39.9/66.0 56.6/48.2 79.49 L 1 2.2/88.0 19.2/78.0 40.1/66.4 58.1/49.6 75.79 FIL-L 1 2.2/ 89.9 20.2/81.7 43.1/71.2 63.2/53.8 66.32 FIL-L 2 2.2/89.4 20.3/82.7 43.9/72.3 65.0/55.2 63.38 L 0 12.1/81.7 59.8/42.8 71.9/20.8 77.9/11.3 41.60 L 1 11.8/79.7 61.2/43.9 74.1/21.4 80.5/11.6 36.70 FIL-L 1 12.4/83.7 66.0/47.3 77.4/22.4 82.9/ 12.0 33.35 FIL-L 2 12.5/84.4 66.1/47.7 78.0/22.5 83.3/12.0 32.30 L 0 11.4/82.6 59.2/45.2 71.8/22.1 77.8/12.0 43.11 L 1 11.2/81.3 60.4/46.2 73.8/22.7 79.9/12.3 38.19 FIL-L 1 11.7/85.1 64.8/49.5 77.2/23.8 82.2/ 12.7 34.76 FIL-L 2 11.6/84.5 65.5/50.0 77.9/24.0 82.7/12.7 34.13 Table 3: The Recall/Precision@ N (%,  ) and mean rank (MR,  ) with different queries on the relevant video retrieval task on the training (top), development (middle) and the test set (bottom). each goal g , we randomly split its relevant videos v g into three sub-sets v tr g , v dev g and v test g with a ratio of 7.5:1.25:1.25, as the training, development, and testing sets. 8 6.2 Setup Since our KB is fully textual, we also represent each video textually with its automatically generated captions. For the search engine, we use Elasticsearch with the standard BM25 metric (Robertson and Zaragoza, 2009). 9 We denote the relevance score calculated by BM25 between the query q and a textually represented video v as Rel ( q, v ) . We experiment with four different methods, which differ in how they construct the query q : L 0 : Goal only. The query is the goal g itself. This is the minimal query without any additional hierarchical information. The relevance score is simply Rel ( q, v ) = Rel ( g, v ) . L 1 : Goal + Children. The query is a concatenation of the goal g and its immediate children steps Ch ( g ) . This query encodes hierarchical knowledge that already exists in wikiHow. The relevance score is then defined as a weighted sum, Rel ( q, v ) = w g Rel ( g, v ) + w s (cid:80) s  Ch ( g ) Rel ( s, v ) . The weights w g and w s are tuned on a development set and set to 1.0 and 0.1 respectively. FIL-L 1 : Goal + Filtered children. The query is a concatenation of the goal g and a filtered sequence of its children Ch ( g ) . Intuitively, decomposing a goal introduces richer information ranked videos likely demonstrate the queried goal. 8 We explain more about the appropriateness of the downstream video retrieval task setup in B.1. but also introduces noise, since certain steps may not visually appear at all ( e.g., enjoy yourself ). Therefore, we perform filtering and only retain the most informative steps, denoted by Ch (cid:48) ( g ) . Specifically, to construct Ch (cid:48) ( g ) for a goal g , we use a hill-climbing algorithm to check each step s from Ch ( g ) , and include s into the query only if it yields better ranking results for the ground-truth videos in the training set v train g . 10 The relevance score is defined as Rel ( q, v ) = w g Rel ( g, v )+ w s (cid:80) s  Ch (cid:48) ( g ) Rel ( s, v ) , where w g is set to 1.0 and w s is set to 0.5 after similar tuning. FIL-L 2 : Goal + Filtered children + Filtered grand-children. The query is the concatenation of the goal g and a filtered sequence of its immediate children Ch ( g ) and grandchildren Ch ( s ) ( s  Ch ( g ) ). These filtered steps are denoted by Ch (cid:48) ( g + Ch ( g )) . This two-level decomposition uses the knowledge from our KB, therefore including lower-level information about the execution of the goal. We perform the same filtering algorithm as in FIL-L 1 , and we define Rel ( q, v ) = w g Rel ( g, v )+ w s (cid:80) s  Ch (cid:48) ( g + Ch ( g )) Rel ( s, v ) . w g is set to 1.0 and w s is set to 0.5. We report the precision@ N , recall@ N and mean rank (MR) following existing work on video retrieval (Luo et al., 2021) (see B.2 for metric def-initions). Table 3 lists the results. First, queries that encode hierarchies of goals ( L 1 , FIL-L 1 and FIL-L 2 ) are generally more beneficial than queries that do not ( L 0 ). The steps of goals enrich a query and assist the retrieval. Second, video-oriented filtering yields significant improvement over the un-filtered L 1 queries since it produces a set of more generalizable steps that are shared among multiple videos. Although steps in wikiHow articles are human-written, they are not grounded to real-world executions of that goal. Many steps do not have corresponding executions in the videos and become noisy steps in the L 1 queries. More interestingly, we observe that queries using deeper hierarchies (FIL-L 2 ) outperform the shallower ones (FIL-L 1 ) in most cases. This is probably due to the fact that how-to videos usually contain detailed (verbal) instructions of a procedure, which are better aligned with more fine-grained steps found in FIL-L 2 . In our qualitative study, we investigate how FIL-L 2 queries with deeper hierarchies help retrieval. Table 4 list FIL-L 1 and FIL-L 2 queries for two goals. We find that the FIL-L 2 queries are more informative and cover more aspects. For example, the FIL-L 2 queries for stain cabinet and make avocado fries consist of the preparation, actual operations, and the post-processing steps, while the FIL-L 1 query only contains the first one. In addition, we search the goals on Google and list the key moments of some randomly sampled videos. 11 These key moments textually describe the important clips of the videos, and therefore they presumably also serve as the query for the goal. We find that the FIL-L 2 query of make avocado fries explains a few necessary steps to accomplish this goal, while the key moment is mostly composed of the ingredients of this dish. This comparison suggests the potential integration of our induced hierarchical knowledge to identify key moments in videos in the future. In this section, we study the properties of the hierarchies. First, what kind of steps are likely to be linked to another goal and are thus decomposed? Second, what do the decomposed steps look like? We group steps into two clusters. The first contains the immediate steps of a goal ( s  Ch ( g ) ) whose prediction is not unlinkable . The second contains the decomposed steps of the steps in the first cluster ( s (cid:48)  Ch ( s ) ). We use spaCy (Hon-nibal et al., 2020) to extract and lemmatize the verb in each step and rank the verbs by their frequency in each cluster. Next, the top100 most frequent verbs in each cluster are selected and we measure the rank difference of these verbs in the two clusters. Figure 3 plots the verbs with largest rank difference and the full figure is in Figure 4. We observe that verbs that convey complex actions and intuitively consist of many other actions become less frequent after the decomposition ( e.g., decorate). On the other hand, verbs that describe the action itself gain in frequency after the decomposition ( e.g., push, hold, press). This observation follows our assump-tion that the decomposition would lead to more fine-grained realizations of a complex procedure. Some other more abstract actions such as learn and decide also increase in frequency, as some low-level goals are explained with more complex steps. Linking Procedural Events To the best of our knowledge, two other pieces of work Pareti et al. (2014); Lagos et al. (2017) tackled the task of linking steps in procedures to other procedures. Both of them also drew the procedures from wikiHow. While we share the same task formulation, our work makes several additional contributions: (1) a retrieval-then-rerank method significantly increases linking recall; (2) more comprehensive experiments with the manual and the downstream evaluation that showcases the quality and usefulness of the linked data and (3) experiments and data with broader coverage over all of WikiHow, not just the Computer domain. Procedural Knowledge Procedural knowledge can be seen as a subset of knowledge pertaining to scripts (Abelson and Schank, 1977; Rudinger et al., 2015), schemata (Rumelhart, 1975) or events. A small body of previous work (Mujtaba and Ma-hapatra, 2019) on procedural events includes extracting them from instructional texts (Paris et al., 2002; Delpech and Saint-Dizier, 2008; Zhang et al., 2012) and videos (Alayrac et al., 2016; Yang et al., 2021a), reasoning about them (Takechi et al., 2003; Tandon et al., 2019; Rajagopal et al., 2020), or showing their downstream applications (Pareti, 2018; Zhang et al., 2020d; Yang et al., 2021b; Zhang et al., 2020b; Lyu et al., 2021), specifically on intent reasoning (Sap et al., 2019; Dalvi et al., 2019; Zhang et al., 2020c). Most procedural datasets are collected by crowdsourcing then manually cleaned (Singh et al., 2002; Regneri et al., 2010; Li et al., 2012; Wanzare et al., 2016; Rashkin et al., 2018) and are hence small. Existing work has also leveraged wikiHow for large-scale knowledge-base construction (Jung et al., 2010; Chu et al., 2017; Park and Motahari Nezhad, 2018), but our work is the first to provide a comprehensive intrinsic and extrinsic evaluation of the resulting knowledge-base. We propose a search-then-rerank algorithm to effectively construct a hierarchical knowledge-base of procedures based on wikiHow. Our hierarchies are shown to help users accomplish tasks by accurately providing decomposition of a step and improve the performance of downstream tasks such as retrieving instructional videos. One interesting extension is to further study and improve the robustness of our two-stage method to tackle more complex linguistic structures of steps and goals ( e.g., negation, conjunction). Another direction is to enrich the resulting knowledge-base by applying our method to other web resources, 12 or to other modalities ( e.g., video clips). Future work 12 e.g., https://www.instructables.com/, https://www.diynet work.com/how-to 3005 could also explore other usages such as comparing and clustering procedures based on their deep hierarchies; or applying the procedural knowledge to control robots in the situated environments. This research is based upon work supported in part by the DARPA KAIROS Program (contract FA8750-19-2-1004), the DARPA LwLL Program (contract FA8750-19-2-0201), the IARPA BETTER Program (contract 2019-19051600004), and the Amazon Alexa Prize TaskBot Competition. Approved for Public Release, Distribution Unlimited. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of Amazon, DARPA, IARPA, or the U.S. Government. We thank Ziyang Li and Ricardo Gonzalez for developing the web demo, John Wieting for support on implementation, and the anonymous crowd workers for their annotations.
abstain abstain abstain method objective result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain objective abstain abstain abstain objective other abstain abstain abstain method abstain abstain method result method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain other abstain other abstain method abstain method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method abstain objective abstain other objective method method abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain abstain abstain objective method abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain method other objective other other other abstain objective result objective result other other other other other other