title: OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature
    Extraction
summary: Vision-Language-Action (VLA) models aim to predict robotic actions based on
  visual observations and language instructions. Existing approaches require
  fine-tuning pre-trained visionlanguage models (VLMs) as visual and language
  features are independently fed into downstream policies, degrading the
  pre-trained semantic alignments. We propose OTTER, a novel VLA architecture
  that leverages these existing alignments through explicit, text-aware visual
  feature extraction. Instead of processing all visual features, OTTER
  selectively extracts and passes only task-relevant visual features that are
  semantically aligned with the language instruction to the policy transformer.
  This allows OTTER to keep the pre-trained vision-language encoders frozen.
  Thereby, OTTER preserves and utilizes the rich semantic understanding learned
  from large-scale pre-training, enabling strong zero-shot generalization
  capabilities. In simulation and real-world experiments, OTTER significantly
  outperforms existing VLA models, demonstrating strong zeroshot generalization
  to novel objects and environments. Video, code, checkpoints, and dataset:
  https://ottervla.github.io/.
paper_authors._id: 67d1bc433c18d49c52723521
paper_authors.name: Huang Huang
paper_authors._id: 67d1bc433c18d49c52723522
paper_authors.name: Fangchen Liu
paper_authors._id: 67d1bc433c18d49c52723523
paper_authors.name: Letian Fu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-12T16:54:28.791Z
paper_authors._id: 64908d6b0c18343a09361b93
paper_authors.avatarUrl: /avatars/2fa44344fe760f68f19de775b76f45b8.svg
paper_authors.fullname: Max (Letian) Fu
paper_authors.type: user
paper_authors.user: mlfu7
paper_authors._id: 67d1bc433c18d49c52723524
paper_authors.name: Tingfan Wu
paper_authors._id: 67d1bc433c18d49c52723525
paper_authors.name: Mustafa Mukadam
paper_authors._id: 67d1bc433c18d49c52723526
paper_authors.name: Jitendra Malik
paper_authors._id: 67d1bc433c18d49c52723527
paper_authors.name: Ken Goldberg
paper_authors._id: 67d1bc433c18d49c52723528
paper_authors.name: Pieter Abbeel
paper_title: OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature
    Extraction
paper_summary: Vision-Language-Action (VLA) models aim to predict robotic actions based on
  visual observations and language instructions. Existing approaches require
  fine-tuning pre-trained visionlanguage models (VLMs) as visual and language
  features are independently fed into downstream policies, degrading the
  pre-trained semantic alignments. We propose OTTER, a novel VLA architecture
  that leverages these existing alignments through explicit, text-aware visual
  feature extraction. Instead of processing all visual features, OTTER
  selectively extracts and passes only task-relevant visual features that are
  semantically aligned with the language instruction to the policy transformer.
  This allows OTTER to keep the pre-trained vision-language encoders frozen.
  Thereby, OTTER preserves and utilizes the rich semantic understanding learned
  from large-scale pre-training, enabling strong zero-shot generalization
  capabilities. In simulation and real-world experiments, OTTER significantly
  outperforms existing VLA models, demonstrating strong zeroshot generalization
  to novel objects and environments. Video, code, checkpoints, and dataset:
  https://ottervla.github.io/.
paper_ai_keywords: VLA models vision-language models (VLMs) policy transformer task-relevant visual features pre-trained vision-language encoders zero-shot generalization simulation real-world experiments zeroshot generalization