title: VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
summary: We explore how reconciling several foundation models (large language models
  and vision-language models) with a novel unified memory mechanism could tackle
  the challenging video understanding problem, especially capturing the long-term
  temporal relations in lengthy videos. In particular, the proposed multimodal
  agent VideoAgent: 1) constructs a structured memory to store both the generic
  temporal event descriptions and object-centric tracking states of the video; 2)
  given an input task query, it employs tools including video segment
  localization and object memory querying along with other visual foundation
  models to interactively solve the task, utilizing the zero-shot tool-use
  ability of LLMs. VideoAgent demonstrates impressive performances on several
  long-horizon video understanding benchmarks, an average increase of 6.6% on
  NExT-QA and 26.0% on EgoSchema over baselines, closing the gap between
  open-sourced models and private counterparts including Gemini 1.5 Pro.
paper_authors._id: 65f9134a1780bc3711558221
paper_authors.name: Yue Fan
paper_authors._id: 65f9134a1780bc3711558222
paper_authors.name: Xiaojian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:59:01.858Z
paper_authors._id: 60dd0e36a15ddd7d2006d2e9
paper_authors.avatarUrl: /avatars/8bd98177a79efbf295be8f6457683297.svg
paper_authors.fullname: Xiaojian Ma
paper_authors.type: user
paper_authors.user: jeasinema
paper_authors._id: 65f9134a1780bc3711558223
paper_authors.name: Rujie Wu
paper_authors._id: 65f9134a1780bc3711558224
paper_authors.name: Yuntao Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:59:26.151Z
paper_authors._id: 64c33dc31920bd98208a244a
paper_authors.avatarUrl: /avatars/a83b691f64c851324336334d0b7b8cc6.svg
paper_authors.fullname: du
paper_authors.type: user
paper_authors.user: yuntao
paper_authors._id: 65f9134a1780bc3711558225
paper_authors.name: Jiaqi Li
paper_authors._id: 65f9134a1780bc3711558226
paper_authors.name: Zhi Gao
paper_authors._id: 65f9134a1780bc3711558227
paper_authors.name: Qing Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-03T08:26:21.818Z
paper_authors._id: 6455f8c2c8f569b995d603c9
paper_authors.avatarUrl: /avatars/3cb2cad3ab123887c270234bb6a4ca43.svg
paper_authors.fullname: Qing Li
paper_authors.type: user
paper_authors.user: li-qing
paper_title: VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
paper_summary: We explore how reconciling several foundation models (large language models
  and vision-language models) with a novel unified memory mechanism could tackle
  the challenging video understanding problem, especially capturing the long-term
  temporal relations in lengthy videos. In particular, the proposed multimodal
  agent VideoAgent: 1) constructs a structured memory to store both the generic
  temporal event descriptions and object-centric tracking states of the video; 2)
  given an input task query, it employs tools including video segment
  localization and object memory querying along with other visual foundation
  models to interactively solve the task, utilizing the zero-shot tool-use
  ability of LLMs. VideoAgent demonstrates impressive performances on several
  long-horizon video understanding benchmarks, an average increase of 6.6% on
  NExT-QA and 26.0% on EgoSchema over baselines, closing the gap between
  open-sourced models and private counterparts including Gemini 1.5 Pro.
paper_ai_summary: A novel unified memory mechanism in VideoAgent enhances video understanding by combining foundation models, improving performance on long-horizon video benchmarks.
paper_ai_keywords: foundation models large language models vision-language models unified memory mechanism video understanding temporal relations structured memory temporal event descriptions object-centric tracking video segment localization object memory querying zero-shot tool-use NExT-QA EgoSchema