title: Can Vision-Language Models Answer Face to Face Questions in the
    Real-World?
summary: AI models have made significant strides in recent years in their ability to
  describe and answer questions about real-world images. They have also made
  progress in the ability to converse with users in real-time using audio input.
  This raises the question: have we reached the point where AI models, connected
  to a camera and microphone, can converse with users in real-time about scenes
  and events that are unfolding live in front of the camera? This has been a
  long-standing goal in AI and is a prerequisite for real-world AI assistants and
  humanoid robots to interact with humans in everyday situations. In this work,
  we introduce a new dataset and benchmark, the Qualcomm Interactive Video
  Dataset (IVD), which allows us to assess the extent to which existing models
  can support these abilities, and to what degree these capabilities can be
  instilled through fine-tuning. The dataset is based on a simple
  question-answering setup, where users ask questions that the system has to
  answer, in real-time, based on the camera and audio input. We show that
  existing models fall far behind human performance on this task, and we identify
  the main sources for the performance gap. However, we also show that for many
  of the required perceptual skills, fine-tuning on this form of data can
  significantly reduce this gap.
paper_authors._id: 67e376caa26895e9a90af7de
paper_authors.name: Reza Pourreza
paper_authors._id: 67e376caa26895e9a90af7df
paper_authors.name: Rishit Dagli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:44:21.889Z
paper_authors._id: 60796959c59d9e1697fa2324
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60796959c59d9e1697fa2324/wxnDm-p3YgB95NV2p4LGF.png
paper_authors.fullname: Rishit Dagli
paper_authors.type: user
paper_authors.user: rishitdagli
paper_authors._id: 67e376caa26895e9a90af7e0
paper_authors.name: Apratim Bhattacharyya
paper_authors._id: 67e376caa26895e9a90af7e1
paper_authors.name: Sunny Panchal
paper_authors._id: 67e376caa26895e9a90af7e2
paper_authors.name: Guillaume Berger
paper_authors._id: 67e376caa26895e9a90af7e3
paper_authors.name: Roland Memisevic
paper_title: Can Vision-Language Models Answer Face to Face Questions in the
    Real-World?
paper_summary: AI models have made significant strides in recent years in their ability to
  describe and answer questions about real-world images. They have also made
  progress in the ability to converse with users in real-time using audio input.
  This raises the question: have we reached the point where AI models, connected
  to a camera and microphone, can converse with users in real-time about scenes
  and events that are unfolding live in front of the camera? This has been a
  long-standing goal in AI and is a prerequisite for real-world AI assistants and
  humanoid robots to interact with humans in everyday situations. In this work,
  we introduce a new dataset and benchmark, the Qualcomm Interactive Video
  Dataset (IVD), which allows us to assess the extent to which existing models
  can support these abilities, and to what degree these capabilities can be
  instilled through fine-tuning. The dataset is based on a simple
  question-answering setup, where users ask questions that the system has to
  answer, in real-time, based on the camera and audio input. We show that
  existing models fall far behind human performance on this task, and we identify
  the main sources for the performance gap. However, we also show that for many
  of the required perceptual skills, fine-tuning on this form of data can
  significantly reduce this gap.
paper_ai_summary: A new dataset and benchmark, Qualcomm Interactive Video Dataset (IVD), assesses existing AI models' real-time interaction capabilities using camera and audio inputs, highlighting gaps and the potential for fine-tuning to improve performance.
paper_ai_keywords: Qualcomm Interactive Video Dataset IVD perceptual skills fine-tuning