AntGPT: Can Large Language Models Help Long-term Action Anticipation
  from Videos?
Can we better anticipate an actor's future actions (e.g. mix eggs) by knowing
what commonly happens after his/her current action (e.g. crack eggs)? What if
we also know the longer-term goal of the actor (e.g. making egg fried rice)?
The long-term action anticipation (LTA) task aims to predict an actor's future
behavior from video observations in the form of verb and noun sequences, and it
is crucial for human-machine interaction. We propose to formulate the LTA task
from two perspectives: a bottom-up approach that predicts the next actions
autoregressively by modeling temporal dynamics; and a top-down approach that
infers the goal of the actor and plans the needed procedure to accomplish the
goal. We hypothesize that large language models (LLMs), which have been
pretrained on procedure text data (e.g. recipes, how-tos), have the potential
to help LTA from both perspectives. It can help provide the prior knowledge on
the possible next actions, and infer the goal given the observed part of a
procedure, respectively. To leverage the LLMs, we propose a two-stage
framework, AntGPT. It first recognizes the actions already performed in the
observed videos and then asks an LLM to predict the future actions via
conditioned generation, or to infer the goal and plan the whole procedure by
chain-of-thought prompting. Empirical results on the Ego4D LTA v1 and v2
benchmarks, EPIC-Kitchens-55, as well as EGTEA GAZE+ demonstrate the
effectiveness of our proposed approach. AntGPT achieves state-of-the-art
performance on all above benchmarks, and can successfully infer the goal and
thus perform goal-conditioned "counterfactual" prediction via qualitative
analysis. Code and model will be released at
https://brown-palm.github.io/AntGPT
64c868867d0ea4e7f11e3224
Qi Zhao
64c868867d0ea4e7f11e3225
Ce Zhang
claimed_verified
2025-05-30T07:00:52.683Z
6519fd9416f0c073c5c551c3
/avatars/597ae2f69e1fa2b4bd32a69d1d999b55.svg
Ce Zhang
user
ceezh
64c868867d0ea4e7f11e3226
Shijie Wang
claimed_verified
2023-08-01T07:44:32.932Z
6438c702c04b3b996ea702fb
/avatars/09a386c4e810193d2b1f0b7799bc8270.svg
Shijie Wang
user
wang-sj16
64c868867d0ea4e7f11e3227
Changcheng Fu
64c868867d0ea4e7f11e3228
Nakul Agarwal
64c868867d0ea4e7f11e3229
Kwonjoon Lee
admin_assigned
2023-08-01T09:43:18.746Z
62a285781d09d5464eea17d4
/avatars/cd8407124e551fd9a0126abd8c3db044.svg
Kwonjoon Lee
user
kjunelee
64c868867d0ea4e7f11e322a
Chen Sun
claimed_verified
2023-08-01T07:44:36.151Z
64b6bac29bdb2adec64b1f86
https://cdn-avatars.huggingface.co/v1/production/uploads/64b6bac29bdb2adec64b1f86/516JDGISCKjtYvaYvprp4.jpeg
Chen Sun
user
jesu9
AntGPT: Can Large Language Models Help Long-term Action Anticipation
  from Videos?
Can we better anticipate an actor's future actions (e.g. mix eggs) by knowing
what commonly happens after his/her current action (e.g. crack eggs)? What if
we also know the longer-term goal of the actor (e.g. making egg fried rice)?
The long-term action anticipation (LTA) task aims to predict an actor's future
behavior from video observations in the form of verb and noun sequences, and it
is crucial for human-machine interaction. We propose to formulate the LTA task
from two perspectives: a bottom-up approach that predicts the next actions
autoregressively by modeling temporal dynamics; and a top-down approach that
infers the goal of the actor and plans the needed procedure to accomplish the
goal. We hypothesize that large language models (LLMs), which have been
pretrained on procedure text data (e.g. recipes, how-tos), have the potential
to help LTA from both perspectives. It can help provide the prior knowledge on
the possible next actions, and infer the goal given the observed part of a
procedure, respectively. To leverage the LLMs, we propose a two-stage
framework, AntGPT. It first recognizes the actions already performed in the
observed videos and then asks an LLM to predict the future actions via
conditioned generation, or to infer the goal and plan the whole procedure by
chain-of-thought prompting. Empirical results on the Ego4D LTA v1 and v2
benchmarks, EPIC-Kitchens-55, as well as EGTEA GAZE+ demonstrate the
effectiveness of our proposed approach. AntGPT achieves state-of-the-art
performance on all above benchmarks, and can successfully infer the goal and
thus perform goal-conditioned "counterfactual" prediction via qualitative
analysis. Code and model will be released at
https://brown-palm.github.io/AntGPT
The proposed AntGPT framework effectively anticipates future actions in videos by integrating large language models to infer goals and predict sequences, achieving state-of-the-art performance on multiple benchmarks.
long-term action anticipation LTA autoregressive prediction temporal dynamics top-down approach large language models LLMs procedure text data conditioned generation chain-of-thought prompting Ego4D EPIC-Kitchens-55 EGTEA GAZE+ goal-conditioned prediction counterfactual prediction