title: RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
summary: We present a scalable, bottom-up and intrinsically diverse data collection
  scheme that can be used for high-level reasoning with long and medium horizons
  and that has 2.2x higher throughput compared to traditional narrow top-down
  step-by-step collection. We collect realistic data by performing any user
  requests within the entirety of 3 office buildings and using multiple robot and
  human embodiments. With this data, we show that models trained on all
  embodiments perform better than ones trained on the robot data only, even when
  evaluated solely on robot episodes. We find that for a fixed collection budget
  it is beneficial to take advantage of cheaper human collection along with robot
  collection. We release a large and highly diverse (29,520 unique instructions)
  dataset dubbed RoboVQA containing 829,502 (video, text) pairs for
  robotics-focused visual question answering. We also demonstrate how evaluating
  real robot experiments with an intervention mechanism enables performing tasks
  to completion, making it deployable with human oversight even if imperfect
  while also providing a single performance metric. We demonstrate a single
  video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is
  capable of performing a variety of grounded high-level reasoning tasks in broad
  realistic settings with a cognitive intervention rate 46% lower than the
  zero-shot state of the art visual language model (VLM) baseline and is able to
  guide real robots through long-horizon tasks. The performance gap with
  zero-shot state-of-the-art models indicates that a lot of grounded data remains
  to be collected for real-world deployment, emphasizing the critical need for
  scalable data collection approaches. Finally, we show that video VLMs
  significantly outperform single-image VLMs with an average error rate reduction
  of 19% across all VQA tasks. Data and videos available at
  https://robovqa.github.io
paper_authors._id: 6544fab22a13610acc1d376f
paper_authors.name: Pierre Sermanet
paper_authors._id: 6544fab22a13610acc1d3770
paper_authors.name: Tianli Ding
paper_authors._id: 6544fab22a13610acc1d3771
paper_authors.name: Jeffrey Zhao
paper_authors._id: 6544fab22a13610acc1d3772
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T08:09:49.294Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 6544fab22a13610acc1d3773
paper_authors.name: Debidatta Dwibedi
paper_authors._id: 6544fab22a13610acc1d3774
paper_authors.name: Keerthana Gopalakrishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-03T16:10:35.935Z
paper_authors._id: 632f7dee34c562cb1d380fdc
paper_authors.avatarUrl: /avatars/d689e260535b8aa775654160028d4ec8.svg
paper_authors.fullname: Keerthana Gopalakrishnan
paper_authors.type: user
paper_authors.user: keerthanpg
paper_authors._id: 6544fab22a13610acc1d3775
paper_authors.name: Christine Chan
paper_authors._id: 6544fab22a13610acc1d3776
paper_authors.name: Gabriel Dulac-Arnold
paper_authors._id: 6544fab22a13610acc1d3777
paper_authors.name: Sharath Maddineni
paper_authors._id: 6544fab22a13610acc1d3778
paper_authors.name: Nikhil J Joshi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-03T16:11:03.603Z
paper_authors._id: 63c15485894342c8964c481d
paper_authors.avatarUrl: /avatars/cd938a12eb73a8395872f4058d85b0c5.svg
paper_authors.fullname: nikhil joshi
paper_authors.type: user
paper_authors.user: valeska
paper_authors._id: 6544fab22a13610acc1d3779
paper_authors.name: Pete Florence
paper_authors._id: 6544fab22a13610acc1d377a
paper_authors.name: Wei Han
paper_authors._id: 6544fab22a13610acc1d377b
paper_authors.name: Robert Baruch
paper_authors._id: 6544fab22a13610acc1d377c
paper_authors.name: Yao Lu
paper_authors._id: 6544fab22a13610acc1d377d
paper_authors.name: Suvir Mirchandani
paper_authors._id: 6544fab22a13610acc1d377e
paper_authors.name: Peng Xu
paper_authors._id: 6544fab22a13610acc1d377f
paper_authors.name: Pannag Sanketi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:49:31.531Z
paper_authors._id: 66b56f73fc66a466e6104bfb
paper_authors.avatarUrl: /avatars/74855aaf580144fafead4b93d6a24a46.svg
paper_authors.fullname: Pannag Sanketi
paper_authors.type: user
paper_authors.user: prsanky
paper_authors._id: 6544fab22a13610acc1d3780
paper_authors.name: Karol Hausman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-03T16:11:36.589Z
paper_authors._id: 64c93e623a3f428da67d9310
paper_authors.avatarUrl: /avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
paper_authors.fullname: Karol Hausman
paper_authors.type: user
paper_authors.user: KarolHausman
paper_authors._id: 6544fab22a13610acc1d3781
paper_authors.name: Izhak Shafran
paper_authors._id: 6544fab22a13610acc1d3782
paper_authors.name: Brian Ichter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-03T16:11:47.187Z
paper_authors._id: 633a26ab474cfeb1a864dc56
paper_authors.avatarUrl: /avatars/cd40accd894fc78810fc0d5108f413e9.svg
paper_authors.fullname: Brian I
paper_authors.type: user
paper_authors.user: brianichter
paper_authors._id: 6544fab22a13610acc1d3783
paper_authors.name: Yuan Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-03T16:12:30.777Z
paper_authors._id: 6433a8c37b8247480105cf7f
paper_authors.avatarUrl: /avatars/a19536f3f3b8eae3609648bea112b381.svg
paper_authors.fullname: Yuan Cao
paper_authors.type: user
paper_authors.user: caoyuan33
paper_title: RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
paper_summary: We present a scalable, bottom-up and intrinsically diverse data collection
  scheme that can be used for high-level reasoning with long and medium horizons
  and that has 2.2x higher throughput compared to traditional narrow top-down
  step-by-step collection. We collect realistic data by performing any user
  requests within the entirety of 3 office buildings and using multiple robot and
  human embodiments. With this data, we show that models trained on all
  embodiments perform better than ones trained on the robot data only, even when
  evaluated solely on robot episodes. We find that for a fixed collection budget
  it is beneficial to take advantage of cheaper human collection along with robot
  collection. We release a large and highly diverse (29,520 unique instructions)
  dataset dubbed RoboVQA containing 829,502 (video, text) pairs for
  robotics-focused visual question answering. We also demonstrate how evaluating
  real robot experiments with an intervention mechanism enables performing tasks
  to completion, making it deployable with human oversight even if imperfect
  while also providing a single performance metric. We demonstrate a single
  video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is
  capable of performing a variety of grounded high-level reasoning tasks in broad
  realistic settings with a cognitive intervention rate 46% lower than the
  zero-shot state of the art visual language model (VLM) baseline and is able to
  guide real robots through long-horizon tasks. The performance gap with
  zero-shot state-of-the-art models indicates that a lot of grounded data remains
  to be collected for real-world deployment, emphasizing the critical need for
  scalable data collection approaches. Finally, we show that video VLMs
  significantly outperform single-image VLMs with an average error rate reduction
  of 19% across all VQA tasks. Data and videos available at
  https://robovqa.github.io
paper_ai_summary: A scalable data collection scheme using both robots and humans within office buildings enhances the performance of models in robotics-focused visual question answering, particularly for long-horizon tasks, and demonstrates the importance of grounded data for real-world deployment.
paper_ai_keywords: data collection high-level reasoning long and medium horizons robot and human embodiments robotics-focused visual question answering RoboVQA dataset video-conditioned model RoboVQA-VideoCoCa cognitive intervention grounded high-level reasoning video VLMs single-image VLMs