Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot
  Manipulation Datasets
Vision-Language Models (VLMs) acquire real-world knowledge and general
reasoning ability through Internet-scale image-text corpora. They can augment
robotic systems with scene understanding and task planning, and assist
visuomotor policies that are trained on robot trajectory data. We explore the
reverse paradigm - using rich, real, multi-modal robot trajectory data to
enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual
Question Answering (VQA) dataset generation framework for VLMs. Given a human
tele-operated robot trajectory, Robo2VLM derives ground-truth from non-visual
and non-descriptive sensory modalities, such as end-effector pose, gripper
aperture, and force sensing. Based on these modalities, it segments the robot
trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses
scene and interaction understanding to identify 3D properties of the robot,
task goal, and the target object. The properties are used to generate
representative VQA queries - images with textural multiple-choice questions -
based on spatial, goal-conditioned, and interaction reasoning question
templates. We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710
questions covering 463 distinct scenes and 3,396 robotic manipulation tasks
from 176k real robot trajectories. Results suggest that Robo2VLM-1 can
benchmark and improve VLM capabilities in spatial and interaction reasoning.
682e8bc5b38184d0edcd1671
Kaiyuan Chen
claimed_verified
2025-05-22T07:16:21.981Z
66d2af23f040611f7cea1b1b
https://cdn-avatars.huggingface.co/v1/production/uploads/66d2af23f040611f7cea1b1b/vCToO_XgrLD0nfR8rzjZd.jpeg
Kaiyuan Eric Chen
user
keplerccc
682e8bc5b38184d0edcd1672
Shuangyu Xie
682e8bc5b38184d0edcd1673
Zehan Ma
682e8bc5b38184d0edcd1674
Ken Goldberg
Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot
  Manipulation Datasets
Vision-Language Models (VLMs) acquire real-world knowledge and general
reasoning ability through Internet-scale image-text corpora. They can augment
robotic systems with scene understanding and task planning, and assist
visuomotor policies that are trained on robot trajectory data. We explore the
reverse paradigm - using rich, real, multi-modal robot trajectory data to
enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual
Question Answering (VQA) dataset generation framework for VLMs. Given a human
tele-operated robot trajectory, Robo2VLM derives ground-truth from non-visual
and non-descriptive sensory modalities, such as end-effector pose, gripper
aperture, and force sensing. Based on these modalities, it segments the robot
trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses
scene and interaction understanding to identify 3D properties of the robot,
task goal, and the target object. The properties are used to generate
representative VQA queries - images with textural multiple-choice questions -
based on spatial, goal-conditioned, and interaction reasoning question
templates. We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710
questions covering 463 distinct scenes and 3,396 robotic manipulation tasks
from 176k real robot trajectories. Results suggest that Robo2VLM-1 can
benchmark and improve VLM capabilities in spatial and interaction reasoning.
Robo2VLM, a framework for generating Visual Question Answering datasets using robot trajectory data, enhances and evaluates Vision-Language Models by leveraging sensory modalities and 3D property understanding.
Visual-Language Models Visual Question Answering VQA robot trajectory data end-effector pose gripper aperture force sensing manipulation phases 3D properties task goal target object spatial reasoning goal-conditioned reasoning interaction reasoning Robo2VLM-1