title: EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in
    the Wild
summary: Predicting when to initiate speech in real-world environments remains a
  fundamental challenge for conversational agents. We introduce EgoSpeak, a novel
  framework for real-time speech initiation prediction in egocentric streaming
  video. By modeling the conversation from the speaker's first-person viewpoint,
  EgoSpeak is tailored for human-like interactions in which a conversational
  agent must continuously observe its environment and dynamically decide when to
  talk. Our approach bridges the gap between simplified experimental setups and
  complex natural conversations by integrating four key capabilities: (1)
  first-person perspective, (2) RGB processing, (3) online processing, and (4)
  untrimmed video processing. We also present YT-Conversation, a diverse
  collection of in-the-wild conversational videos from YouTube, as a resource for
  large-scale pretraining. Experiments on EasyCom and Ego4D demonstrate that
  EgoSpeak outperforms random and silence-based baselines in real time. Our
  results also highlight the importance of multimodal input and context length in
  effectively deciding when to speak.
paper_authors._id: 67bbf87b4f54983efbd94187
paper_authors.name: Junhyeok Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:07:08.715Z
paper_authors._id: 646aecb04c1cd18b497a50ee
paper_authors.avatarUrl: /avatars/de15c724056f36a41cb4f375d05ed836.svg
paper_authors.fullname: Junhyeok Kim
paper_authors.type: user
paper_authors.user: kjunh
paper_authors._id: 67bbf87b4f54983efbd94188
paper_authors.name: Min Soo Kim
paper_authors._id: 67bbf87b4f54983efbd94189
paper_authors.name: Jiwan Chung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T16:10:38.158Z
paper_authors._id: 60d74d1affe0328e0167dc5f
paper_authors.avatarUrl: /avatars/9b1a2df9402e9c26e1eb7c818af9bae0.svg
paper_authors.fullname: Jiwan Chung
paper_authors.type: user
paper_authors.user: jiwan-chung
paper_authors._id: 67bbf87b4f54983efbd9418a
paper_authors.name: Jungbin Cho
paper_authors._id: 67bbf87b4f54983efbd9418b
paper_authors.name: Jisoo Kim
paper_authors._id: 67bbf87b4f54983efbd9418c
paper_authors.name: Sungwoong Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T16:10:11.922Z
paper_authors._id: 662afff1ef7a4675bdf5bfb6
paper_authors.avatarUrl: /avatars/511bc1c630dff30f7651ff8037110792.svg
paper_authors.fullname: sungwoong kim
paper_authors.type: user
paper_authors.user: sukim96
paper_authors._id: 67bbf87b4f54983efbd9418d
paper_authors.name: Gyeongbo Sim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T16:09:47.531Z
paper_authors._id: 673be9ed0a4d127a7f4d0ea6
paper_authors.avatarUrl: /avatars/c15690d22a3b2ce098e195e54c9f414e.svg
paper_authors.fullname: Gyeongbo Sim
paper_authors.type: user
paper_authors.user: gbosim
paper_authors._id: 67bbf87b4f54983efbd9418e
paper_authors.name: Youngjae Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T16:09:38.274Z
paper_authors._id: 6504777fb1da3747a05160c4
paper_authors.avatarUrl: /avatars/b777d98a5ff971ddb4c3e1060bb3e070.svg
paper_authors.fullname: Youngjae Yu
paper_authors.type: user
paper_authors.user: yjyu
paper_title: EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in
    the Wild
paper_summary: Predicting when to initiate speech in real-world environments remains a
  fundamental challenge for conversational agents. We introduce EgoSpeak, a novel
  framework for real-time speech initiation prediction in egocentric streaming
  video. By modeling the conversation from the speaker's first-person viewpoint,
  EgoSpeak is tailored for human-like interactions in which a conversational
  agent must continuously observe its environment and dynamically decide when to
  talk. Our approach bridges the gap between simplified experimental setups and
  complex natural conversations by integrating four key capabilities: (1)
  first-person perspective, (2) RGB processing, (3) online processing, and (4)
  untrimmed video processing. We also present YT-Conversation, a diverse
  collection of in-the-wild conversational videos from YouTube, as a resource for
  large-scale pretraining. Experiments on EasyCom and Ego4D demonstrate that
  EgoSpeak outperforms random and silence-based baselines in real time. Our
  results also highlight the importance of multimodal input and context length in
  effectively deciding when to speak.
paper_ai_keywords: first-person perspective RGB processing online processing untrimmed video processing YT-Conversation EasyCom Ego4D multimodal input context length