title: Video Action Differencing
summary: How do two individuals differ when performing the same action? In this work,
  we introduce Video Action Differencing (VidDiff), the novel task of identifying
  subtle differences between videos of the same action, which has many
  applications, such as coaching and skill learning. To enable development on
  this new task, we first create VidDiffBench, a benchmark dataset containing 549
  video pairs, with human annotations of 4,469 fine-grained action differences
  and 2,075 localization timestamps indicating where these differences occur. Our
  experiments demonstrate that VidDiffBench poses a significant challenge for
  state-of-the-art large multimodal models (LMMs), such as GPT-4o and Qwen2-VL.
  By analyzing failure cases of LMMs on VidDiffBench, we highlight two key
  challenges for this task: localizing relevant sub-actions over two videos and
  fine-grained frame comparison. To overcome these, we propose the VidDiff
  method, an agentic workflow that breaks the task into three stages: action
  difference proposal, keyframe localization, and frame differencing, each stage
  utilizing specialized foundation models. To encourage future research in this
  new task, we release the benchmark at
  https://huggingface.co/datasets/jmhb/VidDiffBench and code at
  http://jmhb0.github.io/viddiff.
paper_authors._id: 67d0e915d0038007e5a75178
paper_authors.name: James Burgess
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:41:30.547Z
paper_authors._id: 650871aeb44445e9b3625c7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650871aeb44445e9b3625c7b/mtx3EnkuNF4z29IosnhaQ.png
paper_authors.fullname: James Burgess
paper_authors.type: user
paper_authors.user: jmhb
paper_authors._id: 67d0e915d0038007e5a75179
paper_authors.name: Xiaohan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:40:56.564Z
paper_authors._id: 65703fab7f50602340d23704
paper_authors.avatarUrl: /avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
paper_authors.fullname: Xiaohan Wang
paper_authors.type: user
paper_authors.user: nicholswang
paper_authors._id: 67d0e915d0038007e5a7517a
paper_authors.name: Yuhui Zhang
paper_authors._id: 67d0e915d0038007e5a7517b
paper_authors.name: Anita Rau
paper_authors._id: 67d0e915d0038007e5a7517c
paper_authors.name: Alejandro Lozano
paper_authors._id: 67d0e915d0038007e5a7517d
paper_authors.name: Lisa Dunlap
paper_authors._id: 67d0e915d0038007e5a7517e
paper_authors.name: Trevor Darrell
paper_authors._id: 67d0e915d0038007e5a7517f
paper_authors.name: Serena Yeung-Levy
paper_title: Video Action Differencing
paper_summary: How do two individuals differ when performing the same action? In this work,
  we introduce Video Action Differencing (VidDiff), the novel task of identifying
  subtle differences between videos of the same action, which has many
  applications, such as coaching and skill learning. To enable development on
  this new task, we first create VidDiffBench, a benchmark dataset containing 549
  video pairs, with human annotations of 4,469 fine-grained action differences
  and 2,075 localization timestamps indicating where these differences occur. Our
  experiments demonstrate that VidDiffBench poses a significant challenge for
  state-of-the-art large multimodal models (LMMs), such as GPT-4o and Qwen2-VL.
  By analyzing failure cases of LMMs on VidDiffBench, we highlight two key
  challenges for this task: localizing relevant sub-actions over two videos and
  fine-grained frame comparison. To overcome these, we propose the VidDiff
  method, an agentic workflow that breaks the task into three stages: action
  difference proposal, keyframe localization, and frame differencing, each stage
  utilizing specialized foundation models. To encourage future research in this
  new task, we release the benchmark at
  https://huggingface.co/datasets/jmhb/VidDiffBench and code at
  http://jmhb0.github.io/viddiff.
paper_ai_keywords: Video Action Differencing (VidDiff) VidDiffBench multimodal models (LMMs) GPT-4o Qwen2-VL action difference proposal keyframe localization frame differencing agentic workflow fine-grained action differences localization timestamps