title: VideoAgent: Long-form Video Understanding with Large Language Model as
    Agent
summary: Long-form video understanding represents a significant challenge within
  computer vision, demanding a model capable of reasoning over long multi-modal
  sequences. Motivated by the human cognitive process for long-form video
  understanding, we emphasize interactive reasoning and planning over the ability
  to process lengthy visual inputs. We introduce a novel agent-based system,
  VideoAgent, that employs a large language model as a central agent to
  iteratively identify and compile crucial information to answer a question, with
  vision-language foundation models serving as tools to translate and retrieve
  visual information. Evaluated on the challenging EgoSchema and NExT-QA
  benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only
  8.4 and 8.2 frames used on average. These results demonstrate superior
  effectiveness and efficiency of our method over the current state-of-the-art
  methods, highlighting the potential of agent-based approaches in advancing
  long-form video understanding.
paper_authors._id: 65f7b20c26f86cf3378c022e
paper_authors.name: Xiaohan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:44:05.420Z
paper_authors._id: 65703fab7f50602340d23704
paper_authors.avatarUrl: /avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
paper_authors.fullname: Xiaohan Wang
paper_authors.type: user
paper_authors.user: nicholswang
paper_authors._id: 65f7b20c26f86cf3378c022f
paper_authors.name: Yuhui Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-18T09:02:09.951Z
paper_authors._id: 62da55164398e21bf7f0e292
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62da55164398e21bf7f0e292/xjKkG8IA2IZZqCdjApSh3.jpeg
paper_authors.fullname: Yuhui Zhang
paper_authors.type: user
paper_authors.user: yuhuizhang
paper_authors._id: 65f7b20c26f86cf3378c0230
paper_authors.name: Orr Zohar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-18T09:02:16.560Z
paper_authors._id: 648c9605565e3a44f3c9bb7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg
paper_authors.fullname: Orr Zohar
paper_authors.type: user
paper_authors.user: orrzohar
paper_authors._id: 65f7b20c26f86cf3378c0231
paper_authors.name: Serena Yeung-Levy
paper_title: VideoAgent: Long-form Video Understanding with Large Language Model as
    Agent
paper_summary: Long-form video understanding represents a significant challenge within
  computer vision, demanding a model capable of reasoning over long multi-modal
  sequences. Motivated by the human cognitive process for long-form video
  understanding, we emphasize interactive reasoning and planning over the ability
  to process lengthy visual inputs. We introduce a novel agent-based system,
  VideoAgent, that employs a large language model as a central agent to
  iteratively identify and compile crucial information to answer a question, with
  vision-language foundation models serving as tools to translate and retrieve
  visual information. Evaluated on the challenging EgoSchema and NExT-QA
  benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only
  8.4 and 8.2 frames used on average. These results demonstrate superior
  effectiveness and efficiency of our method over the current state-of-the-art
  methods, highlighting the potential of agent-based approaches in advancing
  long-form video understanding.
paper_ai_keywords: large language model agent-based system video-language foundation models iterative identification compile crucial information zero-shot accuracy EgoSchema benchmark NExT-QA benchmark interactive reasoning planning multi-modal sequences