EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in
  the Wild
Predicting when to initiate speech in real-world environments remains a
fundamental challenge for conversational agents. We introduce EgoSpeak, a novel
framework for real-time speech initiation prediction in egocentric streaming
video. By modeling the conversation from the speaker's first-person viewpoint,
EgoSpeak is tailored for human-like interactions in which a conversational
agent must continuously observe its environment and dynamically decide when to
talk. Our approach bridges the gap between simplified experimental setups and
complex natural conversations by integrating four key capabilities: (1)
first-person perspective, (2) RGB processing, (3) online processing, and (4)
untrimmed video processing. We also present YT-Conversation, a diverse
collection of in-the-wild conversational videos from YouTube, as a resource for
large-scale pretraining. Experiments on EasyCom and Ego4D demonstrate that
EgoSpeak outperforms random and silence-based baselines in real time. Our
results also highlight the importance of multimodal input and context length in
effectively deciding when to speak.
67bbf87b4f54983efbd94187
Junhyeok Kim
claimed_verified
2025-02-24T09:07:08.715Z
646aecb04c1cd18b497a50ee
/avatars/de15c724056f36a41cb4f375d05ed836.svg
Junhyeok Kim
user
kjunh
67bbf87b4f54983efbd94188
Min Soo Kim
67bbf87b4f54983efbd94189
Jiwan Chung
admin_assigned
2025-02-24T16:10:38.158Z
60d74d1affe0328e0167dc5f
/avatars/9b1a2df9402e9c26e1eb7c818af9bae0.svg
Jiwan Chung
user
jiwan-chung
67bbf87b4f54983efbd9418a
Jungbin Cho
67bbf87b4f54983efbd9418b
Jisoo Kim
67bbf87b4f54983efbd9418c
Sungwoong Kim
admin_assigned
2025-02-24T16:10:11.922Z
662afff1ef7a4675bdf5bfb6
/avatars/511bc1c630dff30f7651ff8037110792.svg
sungwoong kim
user
sukim96
67bbf87b4f54983efbd9418d
Gyeongbo Sim
admin_assigned
2025-02-24T16:09:47.531Z
673be9ed0a4d127a7f4d0ea6
/avatars/c15690d22a3b2ce098e195e54c9f414e.svg
Gyeongbo Sim
user
gbosim
67bbf87b4f54983efbd9418e
Youngjae Yu
admin_assigned
2025-02-24T16:09:38.274Z
6504777fb1da3747a05160c4
/avatars/b777d98a5ff971ddb4c3e1060bb3e070.svg
Youngjae Yu
user
yjyu
EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in
  the Wild
Predicting when to initiate speech in real-world environments remains a
fundamental challenge for conversational agents. We introduce EgoSpeak, a novel
framework for real-time speech initiation prediction in egocentric streaming
video. By modeling the conversation from the speaker's first-person viewpoint,
EgoSpeak is tailored for human-like interactions in which a conversational
agent must continuously observe its environment and dynamically decide when to
talk. Our approach bridges the gap between simplified experimental setups and
complex natural conversations by integrating four key capabilities: (1)
first-person perspective, (2) RGB processing, (3) online processing, and (4)
untrimmed video processing. We also present YT-Conversation, a diverse
collection of in-the-wild conversational videos from YouTube, as a resource for
large-scale pretraining. Experiments on EasyCom and Ego4D demonstrate that
EgoSpeak outperforms random and silence-based baselines in real time. Our
results also highlight the importance of multimodal input and context length in
effectively deciding when to speak.
EgoSpeak is a real-time speech initiation prediction framework that uses an ego-centric perspective and multimodal processing, improving performance over baselines in complex conversational settings.
first-person perspective RGB processing online processing untrimmed video processing YT-Conversation multimodal input context length