title: NavGPT-2: Unleashing Navigational Reasoning Capability for Large
    Vision-Language Models
summary: Capitalizing on the remarkable advancements in Large Language Models (LLMs),
  there is a burgeoning initiative to harness LLMs for instruction following
  robotic navigation. Such a trend underscores the potential of LLMs to
  generalize navigational reasoning and diverse language understanding. However,
  a significant discrepancy in agent performance is observed when integrating
  LLMs in the Vision-and-Language navigation (VLN) tasks compared to previous
  downstream specialist models. Furthermore, the inherent capacity of language to
  interpret and facilitate communication in agent interactions is often
  underutilized in these integrations. In this work, we strive to bridge the
  divide between VLN-specialized models and LLM-based navigation paradigms, while
  maintaining the interpretative prowess of LLMs in generating linguistic
  navigational reasoning. By aligning visual content in a frozen LLM, we
  encompass visual observation comprehension for LLMs and exploit a way to
  incorporate LLMs and navigation policy networks for effective action
  predictions and navigational reasoning. We demonstrate the data efficiency of
  the proposed methods and eliminate the gap between LM-based agents and
  state-of-the-art VLN specialists.
paper_authors._id: 66999347319730566482c9c1
paper_authors.name: Gengze Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:10:35.326Z
paper_authors._id: 6450bcd3673b2bcfaf8681af
paper_authors.avatarUrl: /avatars/f5f93d780562d0772ec5dc1728945fcf.svg
paper_authors.fullname: Gengze Zhou
paper_authors.type: user
paper_authors.user: ZGZzz
paper_authors._id: 66999347319730566482c9c2
paper_authors.name: Yicong Hong
paper_authors._id: 66999347319730566482c9c3
paper_authors.name: Zun Wang
paper_authors._id: 66999347319730566482c9c4
paper_authors.name: Xin Eric Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T19:02:58.197Z
paper_authors._id: 64679a226192d39142245e5e
paper_authors.avatarUrl: /avatars/05abee0b6317f100923936ca2099e9eb.svg
paper_authors.fullname: Xin Eric Wang
paper_authors.type: user
paper_authors.user: xw-eric
paper_authors._id: 66999347319730566482c9c5
paper_authors.name: Qi Wu
paper_title: NavGPT-2: Unleashing Navigational Reasoning Capability for Large
    Vision-Language Models
paper_summary: Capitalizing on the remarkable advancements in Large Language Models (LLMs),
  there is a burgeoning initiative to harness LLMs for instruction following
  robotic navigation. Such a trend underscores the potential of LLMs to
  generalize navigational reasoning and diverse language understanding. However,
  a significant discrepancy in agent performance is observed when integrating
  LLMs in the Vision-and-Language navigation (VLN) tasks compared to previous
  downstream specialist models. Furthermore, the inherent capacity of language to
  interpret and facilitate communication in agent interactions is often
  underutilized in these integrations. In this work, we strive to bridge the
  divide between VLN-specialized models and LLM-based navigation paradigms, while
  maintaining the interpretative prowess of LLMs in generating linguistic
  navigational reasoning. By aligning visual content in a frozen LLM, we
  encompass visual observation comprehension for LLMs and exploit a way to
  incorporate LLMs and navigation policy networks for effective action
  predictions and navigational reasoning. We demonstrate the data efficiency of
  the proposed methods and eliminate the gap between LM-based agents and
  state-of-the-art VLN specialists.
paper_ai_summary: The proposed method bridges the performance gap between Vision-and-Language (VLN) specialist models and Large Language Model (LLM)-based navigation by aligning visual content and integrating LLMs with policy networks for effective navigation.
paper_ai_keywords: Large Language Models LLMs instruction following robotic navigation Vision-and-Language navigation VLN tasks LLM-based navigation linguistic navigational reasoning action predictions navigational reasoning data efficiency