Can Vision-Language Models Answer Face to Face Questions in the
  Real-World?
AI models have made significant strides in recent years in their ability to
describe and answer questions about real-world images. They have also made
progress in the ability to converse with users in real-time using audio input.
This raises the question: have we reached the point where AI models, connected
to a camera and microphone, can converse with users in real-time about scenes
and events that are unfolding live in front of the camera? This has been a
long-standing goal in AI and is a prerequisite for real-world AI assistants and
humanoid robots to interact with humans in everyday situations. In this work,
we introduce a new dataset and benchmark, the Qualcomm Interactive Video
Dataset (IVD), which allows us to assess the extent to which existing models
can support these abilities, and to what degree these capabilities can be
instilled through fine-tuning. The dataset is based on a simple
question-answering setup, where users ask questions that the system has to
answer, in real-time, based on the camera and audio input. We show that
existing models fall far behind human performance on this task, and we identify
the main sources for the performance gap. However, we also show that for many
of the required perceptual skills, fine-tuning on this form of data can
significantly reduce this gap.
67e376caa26895e9a90af7de
Reza Pourreza
67e376caa26895e9a90af7df
Rishit Dagli
claimed_verified
2025-03-26T20:44:21.889Z
60796959c59d9e1697fa2324
https://cdn-avatars.huggingface.co/v1/production/uploads/60796959c59d9e1697fa2324/wxnDm-p3YgB95NV2p4LGF.png
Rishit Dagli
user
rishitdagli
67e376caa26895e9a90af7e0
Apratim Bhattacharyya
67e376caa26895e9a90af7e1
Sunny Panchal
67e376caa26895e9a90af7e2
Guillaume Berger
67e376caa26895e9a90af7e3
Roland Memisevic
Can Vision-Language Models Answer Face to Face Questions in the
  Real-World?
AI models have made significant strides in recent years in their ability to
describe and answer questions about real-world images. They have also made
progress in the ability to converse with users in real-time using audio input.
This raises the question: have we reached the point where AI models, connected
to a camera and microphone, can converse with users in real-time about scenes
and events that are unfolding live in front of the camera? This has been a
long-standing goal in AI and is a prerequisite for real-world AI assistants and
humanoid robots to interact with humans in everyday situations. In this work,
we introduce a new dataset and benchmark, the Qualcomm Interactive Video
Dataset (IVD), which allows us to assess the extent to which existing models
can support these abilities, and to what degree these capabilities can be
instilled through fine-tuning. The dataset is based on a simple
question-answering setup, where users ask questions that the system has to
answer, in real-time, based on the camera and audio input. We show that
existing models fall far behind human performance on this task, and we identify
the main sources for the performance gap. However, we also show that for many
of the required perceptual skills, fine-tuning on this form of data can
significantly reduce this gap.
A new dataset and benchmark, Qualcomm Interactive Video Dataset (IVD), assesses existing AI models' real-time interaction capabilities using camera and audio inputs, highlighting gaps and the potential for fine-tuning to improve performance.
Qualcomm Interactive Video Dataset IVD perceptual skills fine-tuning