ViNT: A Foundation Model for Visual Navigation
General-purpose pre-trained models ("foundation models") have enabled
practitioners to produce generalizable solutions for individual machine
learning problems with datasets that are significantly smaller than those
required for learning from scratch. Such models are typically trained on large
and diverse datasets with weak supervision, consuming much more training data
than is available for any individual downstream application. In this paper, we
describe the Visual Navigation Transformer (ViNT), a foundation model that aims
to bring the success of general-purpose pre-trained models to vision-based
robotic navigation. ViNT is trained with a general goal-reaching objective that
can be used with any navigation dataset, and employs a flexible
Transformer-based architecture to learn navigational affordances and enable
efficient adaptation to a variety of downstream navigational tasks. ViNT is
trained on a number of existing navigation datasets, comprising hundreds of
hours of robotic navigation from a variety of different robotic platforms, and
exhibits positive transfer, outperforming specialist models trained on singular
datasets. ViNT can be augmented with diffusion-based subgoal proposals to
explore novel environments, and can solve kilometer-scale navigation problems
when equipped with long-range heuristics. ViNT can also be adapted to novel
task specifications with a technique inspired by prompt-tuning, where the goal
encoder is replaced by an encoding of another task modality (e.g., GPS
waypoints or routing commands) embedded into the same space of goal tokens.
This flexibility and ability to accommodate a variety of downstream problem
domains establishes ViNT as an effective foundation model for mobile robotics.
For videos, code, and model checkpoints, see our project page at
https://visualnav-transformer.github.io.
649a7a2008c3bee0db52eb67
Dhruv Shah
claimed_verified
2023-06-28T07:41:17.669Z
630478649aef62c4013bbab2
/avatars/23a421be9fb37c0c6446e99adb618d37.svg
Dhruv
user
prieuredesion
649a7a2008c3bee0db52eb68
Ajay Sridhar
649a7a2008c3bee0db52eb69
Nitish Dashora
claimed_verified
2023-06-28T07:41:12.669Z
649b2a57c4afb9f5bcf5309e
/avatars/10f99b870cf1a83fc6d2897ea985363a.svg
Nitish Dashora
user
dashora
649a7a2008c3bee0db52eb6a
Kyle Stachowicz
admin_assigned
2023-06-28T09:03:10.318Z
6307eabda670ed10f9d2571f
/avatars/b0811b25ed4fb7e48dd380898049c764.svg
Kyle Stachowicz
user
kylestach
649a7a2008c3bee0db52eb6b
Kevin Black
admin_assigned
2023-06-28T09:03:43.927Z
646fde2c799a974be31b6031
/avatars/d816aa2d18930f4a634096f1c82a49bf.svg
Kevin Black
user
kvablack
649a7a2008c3bee0db52eb6c
Noriaki Hirose
649a7a2008c3bee0db52eb6d
Sergey Levine
ViNT: A Foundation Model for Visual Navigation
General-purpose pre-trained models ("foundation models") have enabled
practitioners to produce generalizable solutions for individual machine
learning problems with datasets that are significantly smaller than those
required for learning from scratch. Such models are typically trained on large
and diverse datasets with weak supervision, consuming much more training data
than is available for any individual downstream application. In this paper, we
describe the Visual Navigation Transformer (ViNT), a foundation model that aims
to bring the success of general-purpose pre-trained models to vision-based
robotic navigation. ViNT is trained with a general goal-reaching objective that
can be used with any navigation dataset, and employs a flexible
Transformer-based architecture to learn navigational affordances and enable
efficient adaptation to a variety of downstream navigational tasks. ViNT is
trained on a number of existing navigation datasets, comprising hundreds of
hours of robotic navigation from a variety of different robotic platforms, and
exhibits positive transfer, outperforming specialist models trained on singular
datasets. ViNT can be augmented with diffusion-based subgoal proposals to
explore novel environments, and can solve kilometer-scale navigation problems
when equipped with long-range heuristics. ViNT can also be adapted to novel
task specifications with a technique inspired by prompt-tuning, where the goal
encoder is replaced by an encoding of another task modality (e.g., GPS
waypoints or routing commands) embedded into the same space of goal tokens.
This flexibility and ability to accommodate a variety of downstream problem
domains establishes ViNT as an effective foundation model for mobile robotics.
For videos, code, and model checkpoints, see our project page at
https://visualnav-transformer.github.io.
ViNT, a Transformer-based foundation model, excels in vision-based robotic navigation by learning from diverse datasets and adapting to various tasks through techniques like diffusion and prompt-tuning.
Visual Navigation Transformer Transformer-based architecture diffusion-based subgoal proposals prompt-tuning goal encoder goal tokens