ShapeLLM: Universal 3D Object Understanding for Embodied Interaction
<|reference_start|>ShapeLLM: Universal 3D Object Understanding for Embodied Interaction: This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages. ShapeLLM is built upon an improved 3D encoder by extending ReCon to ReCon++ that benefits from multi-view image distillation for enhanced geometry understanding. By utilizing ReCon++ as the 3D point cloud input encoder for LLMs, ShapeLLM is trained on constructed instruction-following data and tested on our newly human-curated benchmark, 3D MM-Vet. ReCon++ and ShapeLLM achieve state-of-the-art performance in 3D geometry understanding and language-unified 3D interaction tasks, such as embodied visual grounding. Project page: https://qizekun.github.io/shapellm/<|reference_end|>
@article{qi2024shapellm:,
    title={ShapeLLM: Universal 3D Object Understanding for Embodied Interaction},
    author={Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng
  Ge, Li Yi, Kaisheng Ma},
    journal={arXiv preprint arXiv:2402.17766},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.17766},
    primaryClass={cs.CV}
}