title: LangNav: Language as a Perceptual Representation for Navigation
summary: We explore the use of language as a perceptual representation for
  vision-and-language navigation. Our approach uses off-the-shelf vision systems
  (for image captioning and object detection) to convert an agent's egocentric
  panoramic view at each time step into natural language descriptions. We then
  finetune a pretrained language model to select an action, based on the current
  view and the trajectory history, that would best fulfill the navigation
  instructions. In contrast to the standard setup which adapts a pretrained
  language model to work directly with continuous visual features from pretrained
  vision models, our approach instead uses (discrete) language as the perceptual
  representation. We explore two use cases of our language-based navigation
  (LangNav) approach on the R2R vision-and-language navigation benchmark:
  generating synthetic trajectories from a prompted large language model (GPT-4)
  with which to finetune a smaller language model; and sim-to-real transfer where
  we transfer a policy learned on a simulated environment (ALFRED) to a
  real-world environment (R2R). Our approach is found to improve upon strong
  baselines that rely on visual features in settings where only a few gold
  trajectories (10-100) are available, demonstrating the potential of using
  language as a perceptual representation for navigation tasks.
paper_authors._id: 652896a6aeb01f59a14f4b41
paper_authors.name: Bowen Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-13T07:52:12.251Z
paper_authors._id: 640e3a753830fd441c2c768d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640e3a753830fd441c2c768d/qztg6ML-c87VD8HajREsH.jpeg
paper_authors.fullname: Bowen Pan
paper_authors.type: user
paper_authors.user: bpan
paper_authors._id: 652896a6aeb01f59a14f4b42
paper_authors.name: Rameswar Panda
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-10-13T01:00:22.575Z
paper_authors._id: 648a7542ac1fbb8a1a654645
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
paper_authors.fullname: Rameswar Panda
paper_authors.type: user
paper_authors.user: rpand002
paper_authors._id: 652896a6aeb01f59a14f4b43
paper_authors.name: SouYoung Jin
paper_authors._id: 652896a6aeb01f59a14f4b44
paper_authors.name: Rogerio Feris
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T08:47:25.977Z
paper_authors._id: 6426053ca5ec4a5cbc516567
paper_authors.avatarUrl: /avatars/7b841e0c58aa746e662fc8c78e3c5537.svg
paper_authors.fullname: Rogerio Feris
paper_authors.type: user
paper_authors.user: rferis
paper_authors._id: 652896a6aeb01f59a14f4b45
paper_authors.name: Aude Oliva
paper_authors._id: 652896a6aeb01f59a14f4b46
paper_authors.name: Phillip Isola
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T08:47:40.566Z
paper_authors._id: 623cb1f3ca87989bbf797eee
paper_authors.avatarUrl: /avatars/13fbd5b123b8676561751f820317600c.svg
paper_authors.fullname: Phillip Isola
paper_authors.type: user
paper_authors.user: phillipi
paper_authors._id: 652896a6aeb01f59a14f4b47
paper_authors.name: Yoon Kim
paper_title: LangNav: Language as a Perceptual Representation for Navigation
paper_summary: We explore the use of language as a perceptual representation for
  vision-and-language navigation. Our approach uses off-the-shelf vision systems
  (for image captioning and object detection) to convert an agent's egocentric
  panoramic view at each time step into natural language descriptions. We then
  finetune a pretrained language model to select an action, based on the current
  view and the trajectory history, that would best fulfill the navigation
  instructions. In contrast to the standard setup which adapts a pretrained
  language model to work directly with continuous visual features from pretrained
  vision models, our approach instead uses (discrete) language as the perceptual
  representation. We explore two use cases of our language-based navigation
  (LangNav) approach on the R2R vision-and-language navigation benchmark:
  generating synthetic trajectories from a prompted large language model (GPT-4)
  with which to finetune a smaller language model; and sim-to-real transfer where
  we transfer a policy learned on a simulated environment (ALFRED) to a
  real-world environment (R2R). Our approach is found to improve upon strong
  baselines that rely on visual features in settings where only a few gold
  trajectories (10-100) are available, demonstrating the potential of using
  language as a perceptual representation for navigation tasks.
paper_ai_summary: Using natural language as a perceptual representation for vision-and-language navigation improves performance over visual feature-based approaches, especially with limited gold trajectories.
paper_ai_keywords: egocentric panoramic view image captioning object detection pretrained language model natural language descriptions vision-and-language navigation R2R benchmark GPT-4 sim-to-real transfer ALFRED environment