title: Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot
    Manipulation Datasets
summary: Vision-Language Models (VLMs) acquire real-world knowledge and general
  reasoning ability through Internet-scale image-text corpora. They can augment
  robotic systems with scene understanding and task planning, and assist
  visuomotor policies that are trained on robot trajectory data. We explore the
  reverse paradigm - using rich, real, multi-modal robot trajectory data to
  enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual
  Question Answering (VQA) dataset generation framework for VLMs. Given a human
  tele-operated robot trajectory, Robo2VLM derives ground-truth from non-visual
  and non-descriptive sensory modalities, such as end-effector pose, gripper
  aperture, and force sensing. Based on these modalities, it segments the robot
  trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses
  scene and interaction understanding to identify 3D properties of the robot,
  task goal, and the target object. The properties are used to generate
  representative VQA queries - images with textural multiple-choice questions -
  based on spatial, goal-conditioned, and interaction reasoning question
  templates. We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710
  questions covering 463 distinct scenes and 3,396 robotic manipulation tasks
  from 176k real robot trajectories. Results suggest that Robo2VLM-1 can
  benchmark and improve VLM capabilities in spatial and interaction reasoning.
paper_authors._id: 682e8bc5b38184d0edcd1671
paper_authors.name: Kaiyuan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:16:21.981Z
paper_authors._id: 66d2af23f040611f7cea1b1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66d2af23f040611f7cea1b1b/vCToO_XgrLD0nfR8rzjZd.jpeg
paper_authors.fullname: Kaiyuan Eric Chen
paper_authors.type: user
paper_authors.user: keplerccc
paper_authors._id: 682e8bc5b38184d0edcd1672
paper_authors.name: Shuangyu Xie
paper_authors._id: 682e8bc5b38184d0edcd1673
paper_authors.name: Zehan Ma
paper_authors._id: 682e8bc5b38184d0edcd1674
paper_authors.name: Ken Goldberg
paper_title: Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot
    Manipulation Datasets
paper_summary: Vision-Language Models (VLMs) acquire real-world knowledge and general
  reasoning ability through Internet-scale image-text corpora. They can augment
  robotic systems with scene understanding and task planning, and assist
  visuomotor policies that are trained on robot trajectory data. We explore the
  reverse paradigm - using rich, real, multi-modal robot trajectory data to
  enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual
  Question Answering (VQA) dataset generation framework for VLMs. Given a human
  tele-operated robot trajectory, Robo2VLM derives ground-truth from non-visual
  and non-descriptive sensory modalities, such as end-effector pose, gripper
  aperture, and force sensing. Based on these modalities, it segments the robot
  trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses
  scene and interaction understanding to identify 3D properties of the robot,
  task goal, and the target object. The properties are used to generate
  representative VQA queries - images with textural multiple-choice questions -
  based on spatial, goal-conditioned, and interaction reasoning question
  templates. We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710
  questions covering 463 distinct scenes and 3,396 robotic manipulation tasks
  from 176k real robot trajectories. Results suggest that Robo2VLM-1 can
  benchmark and improve VLM capabilities in spatial and interaction reasoning.
paper_ai_summary: Robo2VLM, a framework for generating Visual Question Answering datasets using robot trajectory data, enhances and evaluates Vision-Language Models by leveraging sensory modalities and 3D property understanding.
paper_ai_keywords: Visual-Language Models Visual Question Answering VQA robot trajectory data end-effector pose gripper aperture force sensing manipulation phases 3D properties task goal target object spatial reasoning goal-conditioned reasoning interaction reasoning Robo2VLM-1