title: Understanding Co-speech Gestures in-the-wild
summary: Co-speech gestures play a vital role in non-verbal communication. In this
  paper, we introduce a new framework for co-speech gesture understanding in the
  wild. Specifically, we propose three new tasks and benchmarks to evaluate a
  model's capability to comprehend gesture-text-speech associations: (i)
  gesture-based retrieval, (ii) gestured word spotting, and (iii) active speaker
  detection using gestures. We present a new approach that learns a tri-modal
  speech-text-video-gesture representation to solve these tasks. By leveraging a
  combination of global phrase contrastive loss and local gesture-word coupling
  loss, we demonstrate that a strong gesture representation can be learned in a
  weakly supervised manner from videos in the wild. Our learned representations
  outperform previous methods, including large vision-language models (VLMs),
  across all three tasks. Further analysis reveals that speech and text
  modalities capture distinct gesture-related signals, underscoring the
  advantages of learning a shared tri-modal embedding space. The dataset, model,
  and code are available at: https://www.robots.ox.ac.uk/~vgg/research/jegal
paper_authors._id: 67ea65dafdfe44514f1eef1c
paper_authors.name: Sindhu B Hegde
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T12:57:30.227Z
paper_authors._id: 6329b2dabdb6242b42b8e3ce
paper_authors.avatarUrl: /avatars/99c68616b3191862f3ef504851e3fb05.svg
paper_authors.fullname: Sindhu Hegde
paper_authors.type: user
paper_authors.user: sindhuhegde
paper_authors._id: 67ea65dafdfe44514f1eef1d
paper_authors.name: K R Prajwal
paper_authors._id: 67ea65dafdfe44514f1eef1e
paper_authors.name: Taein Kwon
paper_authors._id: 67ea65dafdfe44514f1eef1f
paper_authors.name: Andrew Zisserman
paper_title: Understanding Co-speech Gestures in-the-wild
paper_summary: Co-speech gestures play a vital role in non-verbal communication. In this
  paper, we introduce a new framework for co-speech gesture understanding in the
  wild. Specifically, we propose three new tasks and benchmarks to evaluate a
  model's capability to comprehend gesture-text-speech associations: (i)
  gesture-based retrieval, (ii) gestured word spotting, and (iii) active speaker
  detection using gestures. We present a new approach that learns a tri-modal
  speech-text-video-gesture representation to solve these tasks. By leveraging a
  combination of global phrase contrastive loss and local gesture-word coupling
  loss, we demonstrate that a strong gesture representation can be learned in a
  weakly supervised manner from videos in the wild. Our learned representations
  outperform previous methods, including large vision-language models (VLMs),
  across all three tasks. Further analysis reveals that speech and text
  modalities capture distinct gesture-related signals, underscoring the
  advantages of learning a shared tri-modal embedding space. The dataset, model,
  and code are available at: https://www.robots.ox.ac.uk/~vgg/research/jegal
paper_ai_summary: A new framework for understanding co-speech gestures in real-world scenarios is introduced with three specific tasks and benchmarks, using a tri-modal approach that combines speech, text, and video, and achieves better performance than existing models.
paper_ai_keywords: gesture-based retrieval gestured word spotting active speaker detection tri-modal representation global phrase contrastive loss local gesture-word coupling loss shared tri-modal embedding space