title: Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning
summary: Recent advancements in 3D Large Language Models (3DLLMs) have highlighted
  their potential in building general-purpose agents in the 3D real world, yet
  challenges remain due to the lack of high-quality robust instruction-following
  data, leading to limited discriminative power and generalization of 3DLLMs. In
  this paper, we introduce Robin3D, a powerful 3DLLM trained on large-scale
  instruction-following data generated by our novel data engine, Robust
  Instruction Generation (RIG) engine. RIG generates two key instruction data: 1)
  the Adversarial Instruction-following data, which features mixed negative and
  positive samples to enhance the model's discriminative understanding. 2) the
  Diverse Instruction-following data, which contains various instruction styles
  to enhance model's generalization. As a result, we construct 1 million
  instruction-following data, consisting of 344K Adversarial samples, 508K
  Diverse samples, and 165K benchmark training set samples. To better handle
  these complex instructions, Robin3D first incorporates Relation-Augmented
  Projector to enhance spatial understanding, and then strengthens the object
  referring and grounding ability through ID-Feature Bonding. Robin3D
  consistently outperforms previous methods across five widely-used 3D multimodal
  learning benchmarks, without the need for task-specific fine-tuning. Notably,
  we achieve a 7.8\% improvement in the grounding task (Multi3DRefer) and a 6.9\%
  improvement in the captioning task (Scan2Cap).
paper_authors._id: 66fee5e214b824c86560ce72
paper_authors.name: Weitai Kang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:31:27.875Z
paper_authors._id: 662050a24360f44332caf535
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662050a24360f44332caf535/oVLyKvjCxgVV79taeRJPH.png
paper_authors.fullname: Weitai Kang
paper_authors.type: user
paper_authors.user: weitaikang
paper_authors._id: 66fee5e214b824c86560ce73
paper_authors.name: Haifeng Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T18:00:02.091Z
paper_authors._id: 64d880e2bab152b247103a28
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d880e2bab152b247103a28/2OQV-TO-kGi2XurcdBH5h.jpeg
paper_authors.fullname: Haifeng Huang
paper_authors.type: user
paper_authors.user: ZzZZCHS
paper_authors._id: 66fee5e214b824c86560ce74
paper_authors.name: Yuzhang Shang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T18:00:08.240Z
paper_authors._id: 63a740aec46291f273a95b4c
paper_authors.avatarUrl: /avatars/8c19e639e39f93083ab624747f9fd0b3.svg
paper_authors.fullname: Yuzhang Shang
paper_authors.type: user
paper_authors.user: yuzhang
paper_authors._id: 66fee5e214b824c86560ce75
paper_authors.name: Mubarak Shah
paper_authors._id: 66fee5e214b824c86560ce76
paper_authors.name: Yan Yan
paper_title: Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning
paper_summary: Recent advancements in 3D Large Language Models (3DLLMs) have highlighted
  their potential in building general-purpose agents in the 3D real world, yet
  challenges remain due to the lack of high-quality robust instruction-following
  data, leading to limited discriminative power and generalization of 3DLLMs. In
  this paper, we introduce Robin3D, a powerful 3DLLM trained on large-scale
  instruction-following data generated by our novel data engine, Robust
  Instruction Generation (RIG) engine. RIG generates two key instruction data: 1)
  the Adversarial Instruction-following data, which features mixed negative and
  positive samples to enhance the model's discriminative understanding. 2) the
  Diverse Instruction-following data, which contains various instruction styles
  to enhance model's generalization. As a result, we construct 1 million
  instruction-following data, consisting of 344K Adversarial samples, 508K
  Diverse samples, and 165K benchmark training set samples. To better handle
  these complex instructions, Robin3D first incorporates Relation-Augmented
  Projector to enhance spatial understanding, and then strengthens the object
  referring and grounding ability through ID-Feature Bonding. Robin3D
  consistently outperforms previous methods across five widely-used 3D multimodal
  learning benchmarks, without the need for task-specific fine-tuning. Notably,
  we achieve a 7.8\% improvement in the grounding task (Multi3DRefer) and a 6.9\%
  improvement in the captioning task (Scan2Cap).
paper_ai_summary: Robin3D, a 3D Large Language Model trained on adversarial and diverse instruction-following data, enhances spatial understanding and object referencing, leading to improved performance across multiple 3D learning benchmarks without task-specific fine-tuning.
paper_ai_keywords: 3D Large Language Models 3DLLMs Robust Instruction Generation RIG Adversarial Instruction-following data Diverse Instruction-following data Relation-Augmented Projector ID-Feature Bonding Multi3DRefer Scan2Cap