title: ViNT: A Foundation Model for Visual Navigation
summary: General-purpose pre-trained models ("foundation models") have enabled
  practitioners to produce generalizable solutions for individual machine
  learning problems with datasets that are significantly smaller than those
  required for learning from scratch. Such models are typically trained on large
  and diverse datasets with weak supervision, consuming much more training data
  than is available for any individual downstream application. In this paper, we
  describe the Visual Navigation Transformer (ViNT), a foundation model that aims
  to bring the success of general-purpose pre-trained models to vision-based
  robotic navigation. ViNT is trained with a general goal-reaching objective that
  can be used with any navigation dataset, and employs a flexible
  Transformer-based architecture to learn navigational affordances and enable
  efficient adaptation to a variety of downstream navigational tasks. ViNT is
  trained on a number of existing navigation datasets, comprising hundreds of
  hours of robotic navigation from a variety of different robotic platforms, and
  exhibits positive transfer, outperforming specialist models trained on singular
  datasets. ViNT can be augmented with diffusion-based subgoal proposals to
  explore novel environments, and can solve kilometer-scale navigation problems
  when equipped with long-range heuristics. ViNT can also be adapted to novel
  task specifications with a technique inspired by prompt-tuning, where the goal
  encoder is replaced by an encoding of another task modality (e.g., GPS
  waypoints or routing commands) embedded into the same space of goal tokens.
  This flexibility and ability to accommodate a variety of downstream problem
  domains establishes ViNT as an effective foundation model for mobile robotics.
  For videos, code, and model checkpoints, see our project page at
  https://visualnav-transformer.github.io.
paper_authors._id: 649a7a2008c3bee0db52eb67
paper_authors.name: Dhruv Shah
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T07:41:17.669Z
paper_authors._id: 630478649aef62c4013bbab2
paper_authors.avatarUrl: /avatars/23a421be9fb37c0c6446e99adb618d37.svg
paper_authors.fullname: Dhruv
paper_authors.type: user
paper_authors.user: prieuredesion
paper_authors._id: 649a7a2008c3bee0db52eb68
paper_authors.name: Ajay Sridhar
paper_authors._id: 649a7a2008c3bee0db52eb69
paper_authors.name: Nitish Dashora
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T07:41:12.669Z
paper_authors._id: 649b2a57c4afb9f5bcf5309e
paper_authors.avatarUrl: /avatars/18203f489d8d8d4252eeea00bd6e2f94.svg
paper_authors.fullname: Nitish Dashora
paper_authors.type: user
paper_authors.user: dashora7
paper_authors._id: 649a7a2008c3bee0db52eb6a
paper_authors.name: Kyle Stachowicz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-28T09:03:10.318Z
paper_authors._id: 6307eabda670ed10f9d2571f
paper_authors.avatarUrl: /avatars/b0811b25ed4fb7e48dd380898049c764.svg
paper_authors.fullname: Kyle Stachowicz
paper_authors.type: user
paper_authors.user: kylestach
paper_authors._id: 649a7a2008c3bee0db52eb6b
paper_authors.name: Kevin Black
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-28T09:03:43.927Z
paper_authors._id: 646fde2c799a974be31b6031
paper_authors.avatarUrl: /avatars/d816aa2d18930f4a634096f1c82a49bf.svg
paper_authors.fullname: Kevin Black
paper_authors.type: user
paper_authors.user: kvablack
paper_authors._id: 649a7a2008c3bee0db52eb6c
paper_authors.name: Noriaki Hirose
paper_authors._id: 649a7a2008c3bee0db52eb6d
paper_authors.name: Sergey Levine
paper_title: ViNT: A Foundation Model for Visual Navigation
paper_summary: General-purpose pre-trained models ("foundation models") have enabled
  practitioners to produce generalizable solutions for individual machine
  learning problems with datasets that are significantly smaller than those
  required for learning from scratch. Such models are typically trained on large
  and diverse datasets with weak supervision, consuming much more training data
  than is available for any individual downstream application. In this paper, we
  describe the Visual Navigation Transformer (ViNT), a foundation model that aims
  to bring the success of general-purpose pre-trained models to vision-based
  robotic navigation. ViNT is trained with a general goal-reaching objective that
  can be used with any navigation dataset, and employs a flexible
  Transformer-based architecture to learn navigational affordances and enable
  efficient adaptation to a variety of downstream navigational tasks. ViNT is
  trained on a number of existing navigation datasets, comprising hundreds of
  hours of robotic navigation from a variety of different robotic platforms, and
  exhibits positive transfer, outperforming specialist models trained on singular
  datasets. ViNT can be augmented with diffusion-based subgoal proposals to
  explore novel environments, and can solve kilometer-scale navigation problems
  when equipped with long-range heuristics. ViNT can also be adapted to novel
  task specifications with a technique inspired by prompt-tuning, where the goal
  encoder is replaced by an encoding of another task modality (e.g., GPS
  waypoints or routing commands) embedded into the same space of goal tokens.
  This flexibility and ability to accommodate a variety of downstream problem
  domains establishes ViNT as an effective foundation model for mobile robotics.
  For videos, code, and model checkpoints, see our project page at
  https://visualnav-transformer.github.io.
paper_ai_keywords: Visual Navigation Transformer (ViNT) Transformer-based architecture navigational affordances diffusion-based subgoal proposals long-range heuristics prompt-tuning goal tokens