title: Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to
    Minimize User Input
summary: Existing embodied instance goal navigation tasks, driven by natural language,
  assume human users to provide complete and nuanced instance descriptions prior
  to the navigation, which can be impractical in the real world as human
  instructions might be brief and ambiguous. To bridge this gap, we propose a new
  task, Collaborative Instance Navigation (CoIN), with dynamic agent-human
  interaction during navigation to actively resolve uncertainties about the
  target instance in natural, template-free, open-ended dialogues. To address
  CoIN, we propose a novel method, Agent-user Interaction with UncerTainty
  Awareness (AIUTA), leveraging the perception capability of Vision Language
  Models (VLMs) and the capability of Large Language Models (LLMs). First, upon
  object detection, a Self-Questioner model initiates a self-dialogue to obtain a
  complete and accurate observation description, while a novel uncertainty
  estimation technique mitigates inaccurate VLM perception. Then, an Interaction
  Trigger module determines whether to ask a question to the user, continue or
  halt navigation, minimizing user input. For evaluation, we introduce
  CoIN-Bench, a benchmark supporting both real and simulated humans. AIUTA
  achieves competitive performance in instance navigation against
  state-of-the-art methods, demonstrating great flexibility in handling user
  inputs.
paper_authors._id: 674ec96f8b4bb24d65e3e782
paper_authors.name: Francesco Taioli
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-03T09:05:34.387Z
paper_authors._id: 63ab07a60ed3c32528492782
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ab07a60ed3c32528492782/gLDYEMu39AEWtpy9el-92.jpeg
paper_authors.fullname: Francesco Taioli
paper_authors.type: user
paper_authors.user: ftaioli
paper_authors._id: 674ec96f8b4bb24d65e3e783
paper_authors.name: Edoardo Zorzi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T12:00:11.637Z
paper_authors._id: 6621462611c923d051d62072
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6621462611c923d051d62072/9MEUEZhR-LhrK4XCUck-r.png
paper_authors.fullname: Edoardo
paper_authors.type: user
paper_authors.user: e-zorzi
paper_authors._id: 674ec96f8b4bb24d65e3e784
paper_authors.name: Gianni Franchi
paper_authors._id: 674ec96f8b4bb24d65e3e785
paper_authors.name: Alberto Castellini
paper_authors._id: 674ec96f8b4bb24d65e3e786
paper_authors.name: Alessandro Farinelli
paper_authors._id: 674ec96f8b4bb24d65e3e787
paper_authors.name: Marco Cristani
paper_authors._id: 674ec96f8b4bb24d65e3e788
paper_authors.name: Yiming Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-18T14:41:27.857Z
paper_authors._id: 67d98541faa3f91a1a7491eb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/nzdKQB350VBFZE0QXNLuj.png
paper_authors.fullname: yiming wang
paper_authors.type: user
paper_authors.user: yiming-wang
paper_title: Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to
    Minimize User Input
paper_summary: Existing embodied instance goal navigation tasks, driven by natural language,
  assume human users to provide complete and nuanced instance descriptions prior
  to the navigation, which can be impractical in the real world as human
  instructions might be brief and ambiguous. To bridge this gap, we propose a new
  task, Collaborative Instance Navigation (CoIN), with dynamic agent-human
  interaction during navigation to actively resolve uncertainties about the
  target instance in natural, template-free, open-ended dialogues. To address
  CoIN, we propose a novel method, Agent-user Interaction with UncerTainty
  Awareness (AIUTA), leveraging the perception capability of Vision Language
  Models (VLMs) and the capability of Large Language Models (LLMs). First, upon
  object detection, a Self-Questioner model initiates a self-dialogue to obtain a
  complete and accurate observation description, while a novel uncertainty
  estimation technique mitigates inaccurate VLM perception. Then, an Interaction
  Trigger module determines whether to ask a question to the user, continue or
  halt navigation, minimizing user input. For evaluation, we introduce
  CoIN-Bench, a benchmark supporting both real and simulated humans. AIUTA
  achieves competitive performance in instance navigation against
  state-of-the-art methods, demonstrating great flexibility in handling user
  inputs.
paper_ai_keywords: Collaborative Instance Navigation (CoIN) Agent-user Interaction with UncerTainty Awareness (AIUTA) Vision Language Models (VLMs) Large Language Models (LLMs) Self-Questioner model self-dialogue uncertainty estimation Interaction Trigger module CoIN-Bench