Live Fitness Coaching as a Testbed for Situated Interaction
<|reference_start|>Live Fitness Coaching as a Testbed for Situated Interaction: Tasks at the intersection of vision and language have had a profound impact in advancing the capabilities of vision-language models such as dialog-based assistants. However, models trained on existing tasks are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time are an open challenge. In this work, we present the QEVD benchmark and dataset which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching - a task which intrinsically requires monitoring live user activity and providing timely feedback. It is the first benchmark that requires assistive vision-language models to recognize complex human actions, identify mistakes grounded in those actions, and provide appropriate feedback. Our experiments reveal the limitations of existing state of the art vision-language models for such asynchronous situated interactions. Motivated by this, we propose a simple end-to-end streaming baseline that can respond asynchronously to human actions with appropriate feedbacks at the appropriate time.<|reference_end|>
@article{panchal2024live,
    title={Live Fitness Coaching as a Testbed for Situated Interaction},
    author={Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine
  Mercier, Cornelius Bohm, Florian Dietrichkeit, Reza Pourreza, Xuanlin Li,
  Pulkit Madan, Mingu Lee, Mark Todorovich, Ingo Bax, Roland Memisevic},
    journal={arXiv preprint arXiv:2407.08101},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.08101},
    primaryClass={cs.CV}
}