title: Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D
    Understanding, Generation, and Instruction Following
summary: We introduce Point-Bind, a 3D multi-modality model aligning point clouds with
  2D image, language, audio, and video. Guided by ImageBind, we construct a joint
  embedding space between 3D and multi-modalities, enabling many promising
  applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D
  open-world understanding. On top of this, we further present Point-LLM, the
  first 3D large language model (LLM) following 3D multi-modal instructions. By
  parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of
  Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction
  data, but exhibits superior 3D and multi-modal question-answering capacity. We
  hope our work may cast a light on the community for extending 3D point clouds
  to multi-modality applications. Code is available at
  https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.
paper_authors._id: 64f552773c35e6e18be109d3
paper_authors.name: Ziyu Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-04T09:44:45.787Z
paper_authors._id: 647d9ab61a1fcad2fdbf2d3d
paper_authors.avatarUrl: /avatars/48c8aeae8979d2c87df8bde922437d62.svg
paper_authors.fullname: Ziyu Guo
paper_authors.type: user
paper_authors.user: ZiyuG
paper_authors._id: 64f552773c35e6e18be109d4
paper_authors.name: Renrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-04T12:23:30.820Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 64f552773c35e6e18be109d5
paper_authors.name: Xiangyang Zhu
paper_authors._id: 64f552773c35e6e18be109d6
paper_authors.name: Yiwen Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T08:02:11.225Z
paper_authors._id: 6552f1ad5d55ccb20e9142a0
paper_authors.avatarUrl: /avatars/0e3e80cba64b5ae0bc5638694ac33dbf.svg
paper_authors.fullname: Ivan Tang
paper_authors.type: user
paper_authors.user: IvanTang
paper_authors._id: 64f552773c35e6e18be109d7
paper_authors.name: Xianzheng Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-06T10:49:34.284Z
paper_authors._id: 64f6f80c2d385f3a6d2df467
paper_authors.avatarUrl: /avatars/d5ecfafcbbde40cd785f9d6a28f135e7.svg
paper_authors.fullname: Xianzheng Ma
paper_authors.type: user
paper_authors.user: Oliver-Ma
paper_authors._id: 64f552773c35e6e18be109d8
paper_authors.name: Jiaming Han
paper_authors._id: 64f552773c35e6e18be109d9
paper_authors.name: Kexin Chen
paper_authors._id: 64f552773c35e6e18be109da
paper_authors.name: Peng Gao
paper_authors._id: 64f552773c35e6e18be109db
paper_authors.name: Xianzhi Li
paper_authors._id: 64f552773c35e6e18be109dc
paper_authors.name: Hongsheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:50:19.333Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 64f552773c35e6e18be109dd
paper_authors.name: Pheng-Ann Heng
paper_title: Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D
    Understanding, Generation, and Instruction Following
paper_summary: We introduce Point-Bind, a 3D multi-modality model aligning point clouds with
  2D image, language, audio, and video. Guided by ImageBind, we construct a joint
  embedding space between 3D and multi-modalities, enabling many promising
  applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D
  open-world understanding. On top of this, we further present Point-LLM, the
  first 3D large language model (LLM) following 3D multi-modal instructions. By
  parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of
  Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction
  data, but exhibits superior 3D and multi-modal question-answering capacity. We
  hope our work may cast a light on the community for extending 3D point clouds
  to multi-modality applications. Code is available at
  https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.
paper_ai_keywords: Point-Bind 3D multi language model (LLM) Point-LLM joint embedding space any-to-3D generation 3D embedding arithmetic open-world understanding parameter-efficient fine-tuning 3D instruction multi-modal question-answering capacity LLaMA