Proactive Assistant Dialogue Generation from Streaming Egocentric Videos
Recent advances in conversational AI have been substantial, but developing
real-time systems for perceptual task guidance remains challenging. These
systems must provide interactive, proactive assistance based on streaming
visual inputs, yet their development is constrained by the costly and
labor-intensive process of data collection and system evaluation. To address
these limitations, we present a comprehensive framework with three key
contributions. First, we introduce a novel data curation pipeline that
synthesizes dialogues from annotated egocentric videos, resulting in \dataset,
a large-scale synthetic dialogue dataset spanning multiple domains. Second, we
develop a suite of automatic evaluation metrics, validated through extensive
human studies. Third, we propose an end-to-end model that processes streaming
video inputs to generate contextually appropriate responses, incorporating
novel techniques for handling data imbalance and long-duration videos. This
work lays the foundation for developing real-time, proactive AI assistants
capable of guiding users through diverse tasks. Project page:
https://pro-assist.github.io/
6847e05a3ec10bdd8ab4e03d
Yichi Zhang
extracted_pending
2025-06-10T07:35:55.259Z
6369b1d456d1f93498130a8a
/avatars/8ec228aa6f171715652511f948765db9.svg
Yichi Zhang
user
594zyc
6847e05a3ec10bdd8ab4e03e
Xin Luna Dong
6847e05a3ec10bdd8ab4e03f
Zhaojiang Lin
6847e05a3ec10bdd8ab4e040
Andrea Madotto
6847e05a3ec10bdd8ab4e041
Anuj Kumar
6847e05a3ec10bdd8ab4e042
Babak Damavandi
6847e05a3ec10bdd8ab4e043
Joyce Chai
6847e05a3ec10bdd8ab4e044
Seungwhan Moon
Proactive Assistant Dialogue Generation from Streaming Egocentric Videos
Recent advances in conversational AI have been substantial, but developing
real-time systems for perceptual task guidance remains challenging. These
systems must provide interactive, proactive assistance based on streaming
visual inputs, yet their development is constrained by the costly and
labor-intensive process of data collection and system evaluation. To address
these limitations, we present a comprehensive framework with three key
contributions. First, we introduce a novel data curation pipeline that
synthesizes dialogues from annotated egocentric videos, resulting in \dataset,
a large-scale synthetic dialogue dataset spanning multiple domains. Second, we
develop a suite of automatic evaluation metrics, validated through extensive
human studies. Third, we propose an end-to-end model that processes streaming
video inputs to generate contextually appropriate responses, incorporating
novel techniques for handling data imbalance and long-duration videos. This
work lays the foundation for developing real-time, proactive AI assistants
capable of guiding users through diverse tasks. Project page:
https://pro-assist.github.io/
A framework provides automated data synthesis, evaluation metrics, and an end-to-end model for real-time, proactive conversational AI task guidance using streaming video inputs.
data curation pipeline synthetic dialogue dataset automatic evaluation metrics end-to-end model data imbalance long-duration videos