title: GOAT: GO to Any Thing
summary: In deployment scenarios such as homes and warehouses, mobile robots are
  expected to autonomously navigate for extended periods, seamlessly executing
  tasks articulated in terms that are intuitively understandable by human
  operators. We present GO To Any Thing (GOAT), a universal navigation system
  capable of tackling these requirements with three key features: a) Multimodal:
  it can tackle goals specified via category labels, target images, and language
  descriptions, b) Lifelong: it benefits from its past experience in the same
  environment, and c) Platform Agnostic: it can be quickly deployed on robots
  with different embodiments. GOAT is made possible through a modular system
  design and a continually augmented instance-aware semantic memory that keeps
  track of the appearance of objects from different viewpoints in addition to
  category-level semantics. This enables GOAT to distinguish between different
  instances of the same category to enable navigation to targets specified by
  images and language descriptions. In experimental comparisons spanning over 90
  hours in 9 different homes consisting of 675 goals selected across 200+
  different object instances, we find GOAT achieves an overall success rate of
  83%, surpassing previous methods and ablations by 32% (absolute improvement).
  GOAT improves with experience in the environment, from a 60% success rate at
  the first goal to a 90% success after exploration. In addition, we demonstrate
  that GOAT can readily be applied to downstream tasks such as pick and place and
  social navigation.
paper_authors._id: 6552f3272056e27018cb1101
paper_authors.name: Matthew Chang
paper_authors._id: 6552f3272056e27018cb1102
paper_authors.name: Theophile Gervet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:45:03.548Z
paper_authors._id: 644e822cd6001776ed7700e4
paper_authors.avatarUrl: /avatars/58a72e6bf8ede3cee3de54873231fd3e.svg
paper_authors.fullname: Theophile Gervet
paper_authors.type: user
paper_authors.user: TheophileGervet
paper_authors._id: 6552f3272056e27018cb1103
paper_authors.name: Mukul Khanna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:45:10.582Z
paper_authors._id: 6387a0328b3c55fea3e4a13d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6387a0328b3c55fea3e4a13d/uxMoiqxD0GLKV95ItECJr.jpeg
paper_authors.fullname: Mukul Khanna
paper_authors.type: user
paper_authors.user: mukulkhanna
paper_authors._id: 6552f3272056e27018cb1104
paper_authors.name: Sriram Yenamandra
paper_authors._id: 6552f3272056e27018cb1105
paper_authors.name: Dhruv Shah
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-15T09:21:06.183Z
paper_authors._id: 630478649aef62c4013bbab2
paper_authors.avatarUrl: /avatars/23a421be9fb37c0c6446e99adb618d37.svg
paper_authors.fullname: Dhruv
paper_authors.type: user
paper_authors.user: prieuredesion
paper_authors._id: 6552f3272056e27018cb1106
paper_authors.name: So Yeon Min
paper_authors._id: 6552f3272056e27018cb1107
paper_authors.name: Kavit Shah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:46:14.695Z
paper_authors._id: 649c9efa0c65a5d9697917d4
paper_authors.avatarUrl: /avatars/4c66e4aa278be7709a8b729044583d5d.svg
paper_authors.fullname: Kavit Shah
paper_authors.type: user
paper_authors.user: KShah1998
paper_authors._id: 6552f3272056e27018cb1108
paper_authors.name: Chris Paxton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:46:21.416Z
paper_authors._id: 62b13b28b7053ab38d2d5f96
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b13b28b7053ab38d2d5f96/V6780Omima7sNW5E4TEeW.png
paper_authors.fullname: Chris Paxton
paper_authors.type: user
paper_authors.user: cpaxton
paper_authors._id: 6552f3272056e27018cb1109
paper_authors.name: Saurabh Gupta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-15T09:21:02.788Z
paper_authors._id: 65539ce59075e3304ee69344
paper_authors.avatarUrl: /avatars/af3260f1398477a7d972dc034052bb3b.svg
paper_authors.fullname: Saurabh Gupta
paper_authors.type: user
paper_authors.user: s-gupta
paper_authors._id: 6552f3272056e27018cb110a
paper_authors.name: Dhruv Batra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:46:59.622Z
paper_authors._id: 63ed1e57f765928ceeacbe02
paper_authors.avatarUrl: /avatars/e5a7428bb0acb43a240f1c4a8c1a5639.svg
paper_authors.fullname: Dhruv Batra
paper_authors.type: user
paper_authors.user: dbatra
paper_authors._id: 6552f3272056e27018cb110b
paper_authors.name: Roozbeh Mottaghi
paper_authors._id: 6552f3272056e27018cb110c
paper_authors.name: Jitendra Malik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:47:09.913Z
paper_authors._id: 65369a95605a07338de78ab0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
paper_authors.fullname: Jitendra Malik 
paper_authors.type: user
paper_authors.user: jitendra1995
paper_authors._id: 6552f3272056e27018cb110d
paper_authors.name: Devendra Singh Chaplot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:47:21.169Z
paper_authors._id: 65143e1c4f08b815c8db57a0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65143e1c4f08b815c8db57a0/JqkwKiJmLFRkH0NK3L8XH.jpeg
paper_authors.fullname: Devendra Singh Chaplot
paper_authors.type: user
paper_authors.user: devendrachaplot
paper_title: GOAT: GO to Any Thing
paper_summary: In deployment scenarios such as homes and warehouses, mobile robots are
  expected to autonomously navigate for extended periods, seamlessly executing
  tasks articulated in terms that are intuitively understandable by human
  operators. We present GO To Any Thing (GOAT), a universal navigation system
  capable of tackling these requirements with three key features: a) Multimodal:
  it can tackle goals specified via category labels, target images, and language
  descriptions, b) Lifelong: it benefits from its past experience in the same
  environment, and c) Platform Agnostic: it can be quickly deployed on robots
  with different embodiments. GOAT is made possible through a modular system
  design and a continually augmented instance-aware semantic memory that keeps
  track of the appearance of objects from different viewpoints in addition to
  category-level semantics. This enables GOAT to distinguish between different
  instances of the same category to enable navigation to targets specified by
  images and language descriptions. In experimental comparisons spanning over 90
  hours in 9 different homes consisting of 675 goals selected across 200+
  different object instances, we find GOAT achieves an overall success rate of
  83%, surpassing previous methods and ablations by 32% (absolute improvement).
  GOAT improves with experience in the environment, from a 60% success rate at
  the first goal to a 90% success after exploration. In addition, we demonstrate
  that GOAT can readily be applied to downstream tasks such as pick and place and
  social navigation.
paper_ai_summary: GOAT is a universal navigation system for mobile robots that uses multimodal input, learns from its experience, and is adaptable across different robotic platforms, achieving high success rates in navigation and other tasks.
paper_ai_keywords: multimodal lifelong learning platform agnostic instance-aware semantic memory category-level semantics navigation pick and place social navigation