Understanding Co-speech Gestures in-the-wild
Co-speech gestures play a vital role in non-verbal communication. In this
paper, we introduce a new framework for co-speech gesture understanding in the
wild. Specifically, we propose three new tasks and benchmarks to evaluate a
model's capability to comprehend gesture-text-speech associations: (i)
gesture-based retrieval, (ii) gestured word spotting, and (iii) active speaker
detection using gestures. We present a new approach that learns a tri-modal
speech-text-video-gesture representation to solve these tasks. By leveraging a
combination of global phrase contrastive loss and local gesture-word coupling
loss, we demonstrate that a strong gesture representation can be learned in a
weakly supervised manner from videos in the wild. Our learned representations
outperform previous methods, including large vision-language models (VLMs),
across all three tasks. Further analysis reveals that speech and text
modalities capture distinct gesture-related signals, underscoring the
advantages of learning a shared tri-modal embedding space. The dataset, model,
and code are available at: https://www.robots.ox.ac.uk/~vgg/research/jegal
67ea65dafdfe44514f1eef1c
Sindhu B Hegde
claimed_verified
2025-03-31T12:57:30.227Z
6329b2dabdb6242b42b8e3ce
/avatars/99c68616b3191862f3ef504851e3fb05.svg
Sindhu Hegde
user
sindhuhegde
67ea65dafdfe44514f1eef1d
K R Prajwal
67ea65dafdfe44514f1eef1e
Taein Kwon
67ea65dafdfe44514f1eef1f
Andrew Zisserman
Understanding Co-speech Gestures in-the-wild
Co-speech gestures play a vital role in non-verbal communication. In this
paper, we introduce a new framework for co-speech gesture understanding in the
wild. Specifically, we propose three new tasks and benchmarks to evaluate a
model's capability to comprehend gesture-text-speech associations: (i)
gesture-based retrieval, (ii) gestured word spotting, and (iii) active speaker
detection using gestures. We present a new approach that learns a tri-modal
speech-text-video-gesture representation to solve these tasks. By leveraging a
combination of global phrase contrastive loss and local gesture-word coupling
loss, we demonstrate that a strong gesture representation can be learned in a
weakly supervised manner from videos in the wild. Our learned representations
outperform previous methods, including large vision-language models (VLMs),
across all three tasks. Further analysis reveals that speech and text
modalities capture distinct gesture-related signals, underscoring the
advantages of learning a shared tri-modal embedding space. The dataset, model,
and code are available at: https://www.robots.ox.ac.uk/~vgg/research/jegal
A new framework for understanding co-speech gestures in real-world scenarios is introduced with three specific tasks and benchmarks, using a tri-modal approach that combines speech, text, and video, and achieves better performance than existing models.
gesture-based retrieval gestured word spotting active speaker detection tri-modal representation global phrase contrastive loss local gesture-word coupling loss shared tri-modal embedding space