title: AntGPT: Can Large Language Models Help Long-term Action Anticipation
    from Videos?
summary: Can we better anticipate an actor's future actions (e.g. mix eggs) by knowing
  what commonly happens after his/her current action (e.g. crack eggs)? What if
  we also know the longer-term goal of the actor (e.g. making egg fried rice)?
  The long-term action anticipation (LTA) task aims to predict an actor's future
  behavior from video observations in the form of verb and noun sequences, and it
  is crucial for human-machine interaction. We propose to formulate the LTA task
  from two perspectives: a bottom-up approach that predicts the next actions
  autoregressively by modeling temporal dynamics; and a top-down approach that
  infers the goal of the actor and plans the needed procedure to accomplish the
  goal. We hypothesize that large language models (LLMs), which have been
  pretrained on procedure text data (e.g. recipes, how-tos), have the potential
  to help LTA from both perspectives. It can help provide the prior knowledge on
  the possible next actions, and infer the goal given the observed part of a
  procedure, respectively. To leverage the LLMs, we propose a two-stage
  framework, AntGPT. It first recognizes the actions already performed in the
  observed videos and then asks an LLM to predict the future actions via
  conditioned generation, or to infer the goal and plan the whole procedure by
  chain-of-thought prompting. Empirical results on the Ego4D LTA v1 and v2
  benchmarks, EPIC-Kitchens-55, as well as EGTEA GAZE+ demonstrate the
  effectiveness of our proposed approach. AntGPT achieves state-of-the-art
  performance on all above benchmarks, and can successfully infer the goal and
  thus perform goal-conditioned "counterfactual" prediction via qualitative
  analysis. Code and model will be released at
  https://brown-palm.github.io/AntGPT
paper_authors._id: 64c868867d0ea4e7f11e3224
paper_authors.name: Qi Zhao
paper_authors._id: 64c868867d0ea4e7f11e3225
paper_authors.name: Ce Zhang
paper_authors._id: 64c868867d0ea4e7f11e3226
paper_authors.name: Shijie Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T07:44:32.932Z
paper_authors._id: 6438c702c04b3b996ea702fb
paper_authors.avatarUrl: /avatars/09a386c4e810193d2b1f0b7799bc8270.svg
paper_authors.fullname: Shijie Wang
paper_authors.type: user
paper_authors.user: wang-sj16
paper_authors._id: 64c868867d0ea4e7f11e3227
paper_authors.name: Changcheng Fu
paper_authors._id: 64c868867d0ea4e7f11e3228
paper_authors.name: Nakul Agarwal
paper_authors._id: 64c868867d0ea4e7f11e3229
paper_authors.name: Kwonjoon Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:43:18.746Z
paper_authors._id: 62a285781d09d5464eea17d4
paper_authors.avatarUrl: /avatars/cd8407124e551fd9a0126abd8c3db044.svg
paper_authors.fullname: Kwonjoon Lee
paper_authors.type: user
paper_authors.user: kjunelee
paper_authors._id: 64c868867d0ea4e7f11e322a
paper_authors.name: Chen Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T07:44:36.151Z
paper_authors._id: 64b6bac29bdb2adec64b1f86
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b6bac29bdb2adec64b1f86/516JDGISCKjtYvaYvprp4.jpeg
paper_authors.fullname: Chen Sun
paper_authors.type: user
paper_authors.user: jesu9
paper_title: AntGPT: Can Large Language Models Help Long-term Action Anticipation
    from Videos?
paper_summary: Can we better anticipate an actor's future actions (e.g. mix eggs) by knowing
  what commonly happens after his/her current action (e.g. crack eggs)? What if
  we also know the longer-term goal of the actor (e.g. making egg fried rice)?
  The long-term action anticipation (LTA) task aims to predict an actor's future
  behavior from video observations in the form of verb and noun sequences, and it
  is crucial for human-machine interaction. We propose to formulate the LTA task
  from two perspectives: a bottom-up approach that predicts the next actions
  autoregressively by modeling temporal dynamics; and a top-down approach that
  infers the goal of the actor and plans the needed procedure to accomplish the
  goal. We hypothesize that large language models (LLMs), which have been
  pretrained on procedure text data (e.g. recipes, how-tos), have the potential
  to help LTA from both perspectives. It can help provide the prior knowledge on
  the possible next actions, and infer the goal given the observed part of a
  procedure, respectively. To leverage the LLMs, we propose a two-stage
  framework, AntGPT. It first recognizes the actions already performed in the
  observed videos and then asks an LLM to predict the future actions via
  conditioned generation, or to infer the goal and plan the whole procedure by
  chain-of-thought prompting. Empirical results on the Ego4D LTA v1 and v2
  benchmarks, EPIC-Kitchens-55, as well as EGTEA GAZE+ demonstrate the
  effectiveness of our proposed approach. AntGPT achieves state-of-the-art
  performance on all above benchmarks, and can successfully infer the goal and
  thus perform goal-conditioned "counterfactual" prediction via qualitative
  analysis. Code and model will be released at
  https://brown-palm.github.io/AntGPT
paper_ai_keywords: LTA (long-term action anticipation) autoregressive prediction temporal dynamics large language models (LLMs) procedure text data recipes how-tos prior knowledge goal inference chain-of-thought prompting Ego4D LTA v1 and v2 benchmarks EPIC-Kitchens-55 EGTEA GAZE+ conditioned generation goal-conditioned prediction counterfactual prediction