title: VideoWebArena: Evaluating Long Context Multimodal Agents with Video
    Understanding Web Tasks
summary: Videos are often used to learn or extract the necessary information to
  complete tasks in ways different than what text and static imagery alone can
  provide. However, many existing agent benchmarks neglect long-context video
  understanding, instead focusing on text or static image inputs. To bridge this
  gap, we introduce VideoWebArena (VideoWA), a benchmark for evaluating the
  capabilities of long-context multimodal agents for video understanding. VideoWA
  consists of 2,021 web agent tasks based on manually crafted video tutorials,
  which total almost four hours of content. For our benchmark, we define a
  taxonomy of long-context video-based agent tasks with two main areas of focus:
  skill retention and factual retention. While skill retention tasks evaluate
  whether an agent can use a given human demonstration to complete a task
  efficiently, the factual retention task evaluates whether an agent can retrieve
  instruction-relevant information from a video to complete a task. We find that
  the best model achieves 13.3% success on factual retention tasks and 45.8% on
  factual retention QA pairs, far below human performance at 73.9% and 79.3%,
  respectively. On skill retention tasks, long-context models perform worse with
  tutorials than without, exhibiting a 5% performance decrease in WebArena tasks
  and a 10.3% decrease in VisualWebArena tasks. Our work highlights the need to
  improve the agentic abilities of long-context multimodal models and provides a
  testbed for future development with long-context video agents.
paper_authors._id: 672113b370842ddb5b173250
paper_authors.name: Lawrence Jang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:34:16.391Z
paper_authors._id: 664aebe829eadb3ab4e4ca3f
paper_authors.avatarUrl: /avatars/548d5656e082c8959ac78b883f0805af.svg
paper_authors.fullname: Lawrence Jang
paper_authors.type: user
paper_authors.user: ljang0
paper_authors._id: 672113b370842ddb5b173251
paper_authors.name: Yinheng Li
paper_authors._id: 672113b370842ddb5b173252
paper_authors.name: Charles Ding
paper_authors._id: 672113b370842ddb5b173253
paper_authors.name: Justin Lin
paper_authors._id: 672113b370842ddb5b173254
paper_authors.name: Paul Pu Liang
paper_authors._id: 672113b370842ddb5b173255
paper_authors.name: Dan Zhao
paper_authors._id: 672113b370842ddb5b173256
paper_authors.name: Rogerio Bonatti
paper_authors._id: 672113b370842ddb5b173257
paper_authors.name: Kazuhito Koishida
paper_title: VideoWebArena: Evaluating Long Context Multimodal Agents with Video
    Understanding Web Tasks
paper_summary: Videos are often used to learn or extract the necessary information to
  complete tasks in ways different than what text and static imagery alone can
  provide. However, many existing agent benchmarks neglect long-context video
  understanding, instead focusing on text or static image inputs. To bridge this
  gap, we introduce VideoWebArena (VideoWA), a benchmark for evaluating the
  capabilities of long-context multimodal agents for video understanding. VideoWA
  consists of 2,021 web agent tasks based on manually crafted video tutorials,
  which total almost four hours of content. For our benchmark, we define a
  taxonomy of long-context video-based agent tasks with two main areas of focus:
  skill retention and factual retention. While skill retention tasks evaluate
  whether an agent can use a given human demonstration to complete a task
  efficiently, the factual retention task evaluates whether an agent can retrieve
  instruction-relevant information from a video to complete a task. We find that
  the best model achieves 13.3% success on factual retention tasks and 45.8% on
  factual retention QA pairs, far below human performance at 73.9% and 79.3%,
  respectively. On skill retention tasks, long-context models perform worse with
  tutorials than without, exhibiting a 5% performance decrease in WebArena tasks
  and a 10.3% decrease in VisualWebArena tasks. Our work highlights the need to
  improve the agentic abilities of long-context multimodal models and provides a
  testbed for future development with long-context video agents.
paper_ai_summary: A benchmark named VideoWebArena evaluates long-context video understanding in multimodal agents, showing that current models perform significantly worse than humans in both skill and factual retention tasks.
paper_ai_keywords: long-context multimodal agents video understanding VideoWebArena manual video tutorials skill retention factual retention agentic abilities long-context models WebArena VisualWebArena