title: Learning to Ground Instructional Articles in Videos through Narrations
summary: In this paper we present an approach for localizing steps of procedural
  activities in narrated how-to videos. To deal with the scarcity of labeled data
  at scale, we source the step descriptions from a language knowledge base
  (wikiHow) containing instructional articles for a large variety of procedural
  tasks. Without any form of manual supervision, our model learns to temporally
  ground the steps of procedural articles in how-to videos by matching three
  modalities: frames, narrations, and step descriptions. Specifically, our method
  aligns steps to video by fusing information from two distinct pathways: i) {\em
  direct} alignment of step descriptions to frames, ii) {\em indirect} alignment
  obtained by composing steps-to-narrations with narrations-to-video
  correspondences. Notably, our approach performs global temporal grounding of
  all steps in an article at once by exploiting order information, and is trained
  with step pseudo-labels which are iteratively refined and aggressively
  filtered. In order to validate our model we introduce a new evaluation
  benchmark -- HT-Step -- obtained by manually annotating a 124-hour subset of
  HowTo100MA test server is accessible at
  \url{https://eval.ai/web/challenges/challenge-page/2082.} with steps sourced
  from wikiHow articles. Experiments on this benchmark as well as zero-shot
  evaluations on CrossTask demonstrate that our multi-modality alignment yields
  dramatic gains over several baselines and prior works. Finally, we show that
  our inner module for matching narration-to-video outperforms by a large margin
  the state of the art on the HTM-Align narration-video alignment benchmark.
paper_authors._id: 647fe7c7d037515be11641d7
paper_authors.name: Effrosyni Mavroudi
paper_authors._id: 647fe7c7d037515be11641d8
paper_authors.name: Triantafyllos Afouras
paper_authors._id: 647fe7c7d037515be11641d9
paper_authors.name: Lorenzo Torresani
paper_title: Learning to Ground Instructional Articles in Videos through Narrations
paper_summary: In this paper we present an approach for localizing steps of procedural
  activities in narrated how-to videos. To deal with the scarcity of labeled data
  at scale, we source the step descriptions from a language knowledge base
  (wikiHow) containing instructional articles for a large variety of procedural
  tasks. Without any form of manual supervision, our model learns to temporally
  ground the steps of procedural articles in how-to videos by matching three
  modalities: frames, narrations, and step descriptions. Specifically, our method
  aligns steps to video by fusing information from two distinct pathways: i) {\em
  direct} alignment of step descriptions to frames, ii) {\em indirect} alignment
  obtained by composing steps-to-narrations with narrations-to-video
  correspondences. Notably, our approach performs global temporal grounding of
  all steps in an article at once by exploiting order information, and is trained
  with step pseudo-labels which are iteratively refined and aggressively
  filtered. In order to validate our model we introduce a new evaluation
  benchmark -- HT-Step -- obtained by manually annotating a 124-hour subset of
  HowTo100MA test server is accessible at
  \url{https://eval.ai/web/challenges/challenge-page/2082.} with steps sourced
  from wikiHow articles. Experiments on this benchmark as well as zero-shot
  evaluations on CrossTask demonstrate that our multi-modality alignment yields
  dramatic gains over several baselines and prior works. Finally, we show that
  our inner module for matching narration-to-video outperforms by a large margin
  the state of the art on the HTM-Align narration-video alignment benchmark.
paper_ai_keywords: procedural activities narrated how-to videos language knowledge base step descriptions wikiHow procedural tasks temporal grounding modality frames narrations step descriptions direct alignment indirect alignment steps-to-narrations narrations-to-video correspondences global temporal grounding order information step pseudo-labels evaluation benchmark HT-Step zero-shot evaluations CrossTask multi-modality alignment narration-to-video HTM-Align