title: Multiscale Video Pretraining for Long-Term Activity Forecasting
summary: Long-term activity forecasting is an especially challenging research problem
  because it requires understanding the temporal relationships between observed
  actions, as well as the variability and complexity of human activities. Despite
  relying on strong supervision via expensive human annotations, state-of-the-art
  forecasting approaches often generalize poorly to unseen data. To alleviate
  this issue, we propose Multiscale Video Pretraining (MVP), a novel
  self-supervised pretraining approach that learns robust representations for
  forecasting by learning to predict contextualized representations of future
  video clips over multiple timescales. MVP is based on our observation that
  actions in videos have a multiscale nature, where atomic actions typically
  occur at a short timescale and more complex actions may span longer timescales.
  We compare MVP to state-of-the-art self-supervised video learning approaches on
  downstream long-term forecasting tasks including long-term action anticipation
  and video summary prediction. Our comprehensive experiments across the Ego4D
  and Epic-Kitchens-55/100 datasets demonstrate that MVP out-performs
  state-of-the-art methods by significant margins. Notably, MVP obtains a
  relative performance gain of over 20% accuracy in video summary forecasting
  over existing methods.
paper_authors._id: 64bf90e5a0e54710669129bc
paper_authors.name: Reuben Tan
paper_authors._id: 64bf90e5a0e54710669129bd
paper_authors.name: Matthias De Lange
paper_authors._id: 64bf90e5a0e54710669129be
paper_authors.name: Michael Iuzzolino
paper_authors._id: 64bf90e5a0e54710669129bf
paper_authors.name: Bryan A. Plummer
paper_authors._id: 64bf90e5a0e54710669129c0
paper_authors.name: Kate Saenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T09:19:06.958Z
paper_authors._id: 6272e87b72307af646a5372c
paper_authors.avatarUrl: /avatars/258f3977ae88d747892dcf83b2c56c2e.svg
paper_authors.fullname: Kate Saenko
paper_authors.type: user
paper_authors.user: KateSaenko
paper_authors._id: 64bf90e5a0e54710669129c1
paper_authors.name: Karl Ridgeway
paper_authors._id: 64bf90e5a0e54710669129c2
paper_authors.name: Lorenzo Torresani
paper_title: Multiscale Video Pretraining for Long-Term Activity Forecasting
paper_summary: Long-term activity forecasting is an especially challenging research problem
  because it requires understanding the temporal relationships between observed
  actions, as well as the variability and complexity of human activities. Despite
  relying on strong supervision via expensive human annotations, state-of-the-art
  forecasting approaches often generalize poorly to unseen data. To alleviate
  this issue, we propose Multiscale Video Pretraining (MVP), a novel
  self-supervised pretraining approach that learns robust representations for
  forecasting by learning to predict contextualized representations of future
  video clips over multiple timescales. MVP is based on our observation that
  actions in videos have a multiscale nature, where atomic actions typically
  occur at a short timescale and more complex actions may span longer timescales.
  We compare MVP to state-of-the-art self-supervised video learning approaches on
  downstream long-term forecasting tasks including long-term action anticipation
  and video summary prediction. Our comprehensive experiments across the Ego4D
  and Epic-Kitchens-55/100 datasets demonstrate that MVP out-performs
  state-of-the-art methods by significant margins. Notably, MVP obtains a
  relative performance gain of over 20% accuracy in video summary forecasting
  over existing methods.
paper_ai_keywords: Multiscale Video Pretraining (MVP) contextualized representations long-term action anticipation video summary prediction Ego4D Epic-Kitchens-55/100 datasets self-supervised pretraining approach robust pretrained features