title: Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous
    Sensors via Language Grounding
summary: Interacting with the world is a multi-sensory experience: achieving effective
  general-purpose interaction requires making use of all available modalities --
  including vision, touch, and audio -- to fill in gaps from partial observation.
  For example, when vision is occluded reaching into a bag, a robot should rely
  on its senses of touch and sound. However, state-of-the-art generalist robot
  policies are typically trained on large datasets to predict robot actions
  solely from visual and proprioceptive observations. In this work, we propose
  FuSe, a novel approach that enables finetuning visuomotor generalist policies
  on heterogeneous sensor modalities for which large datasets are not readily
  available by leveraging natural language as a common cross-modal grounding. We
  combine a multimodal contrastive loss with a sensory-grounded language
  generation loss to encode high-level semantics. In the context of robot
  manipulation, we show that FuSe enables performing challenging tasks that
  require reasoning jointly over modalities such as vision, touch, and sound in a
  zero-shot setting, such as multimodal prompting, compositional cross-modal
  prompting, and descriptions of objects it interacts with. We show that the same
  recipe is applicable to widely different generalist policies, including both
  diffusion-based generalist policies and large vision-language-action (VLA)
  models. Extensive experiments in the real world show that FuSeis able to
  increase success rates by over 20% compared to all considered baselines.
paper_authors._id: 6786a3632631abf6966ce890
paper_authors.name: Joshua Jones
paper_authors._id: 6786a3632631abf6966ce891
paper_authors.name: Oier Mees
paper_authors._id: 6786a3632631abf6966ce892
paper_authors.name: Carmelo Sferrazza
paper_authors._id: 6786a3632631abf6966ce893
paper_authors.name: Kyle Stachowicz
paper_authors._id: 6786a3632631abf6966ce894
paper_authors.name: Pieter Abbeel
paper_authors._id: 6786a3632631abf6966ce895
paper_authors.name: Sergey Levine
paper_title: Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous
    Sensors via Language Grounding
paper_summary: Interacting with the world is a multi-sensory experience: achieving effective
  general-purpose interaction requires making use of all available modalities --
  including vision, touch, and audio -- to fill in gaps from partial observation.
  For example, when vision is occluded reaching into a bag, a robot should rely
  on its senses of touch and sound. However, state-of-the-art generalist robot
  policies are typically trained on large datasets to predict robot actions
  solely from visual and proprioceptive observations. In this work, we propose
  FuSe, a novel approach that enables finetuning visuomotor generalist policies
  on heterogeneous sensor modalities for which large datasets are not readily
  available by leveraging natural language as a common cross-modal grounding. We
  combine a multimodal contrastive loss with a sensory-grounded language
  generation loss to encode high-level semantics. In the context of robot
  manipulation, we show that FuSe enables performing challenging tasks that
  require reasoning jointly over modalities such as vision, touch, and sound in a
  zero-shot setting, such as multimodal prompting, compositional cross-modal
  prompting, and descriptions of objects it interacts with. We show that the same
  recipe is applicable to widely different generalist policies, including both
  diffusion-based generalist policies and large vision-language-action (VLA)
  models. Extensive experiments in the real world show that FuSeis able to
  increase success rates by over 20% compared to all considered baselines.
paper_ai_keywords: visuomotor heterogeneous sensor modalities natural language cross-modal grounding multimodal contrastive loss sensory-grounded language generation loss high-level semantics robot manipulation zero-shot setting multimodal prompting compositional cross-modal prompting description of objects diffusion-based generalist policies vision-language-action (VLA) models success rates baselines