Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D
  Understanding, Generation, and Instruction Following
We introduce Point-Bind, a 3D multi-modality model aligning point clouds with
2D image, language, audio, and video. Guided by ImageBind, we construct a joint
embedding space between 3D and multi-modalities, enabling many promising
applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D
open-world understanding. On top of this, we further present Point-LLM, the
first 3D large language model (LLM) following 3D multi-modal instructions. By
parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of
Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction
data, but exhibits superior 3D and multi-modal question-answering capacity. We
hope our work may cast a light on the community for extending 3D point clouds
to multi-modality applications. Code is available at
https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.
64f552773c35e6e18be109d3
Ziyu Guo
admin_assigned
2023-09-04T09:44:45.787Z
647d9ab61a1fcad2fdbf2d3d
/avatars/48c8aeae8979d2c87df8bde922437d62.svg
Ziyu Guo
user
ZiyuG
64f552773c35e6e18be109d4
Renrui Zhang
claimed_verified
2023-09-04T12:23:30.820Z
645b8b2687c79b6ec0bb3b7a
/avatars/00a9db32a42dc950112bf2593bb109cb.svg
Renrui
user
ZrrSkywalker
64f552773c35e6e18be109d5
Xiangyang Zhu
64f552773c35e6e18be109d6
Yiwen Tang
claimed_verified
2025-02-14T08:02:11.225Z
6552f1ad5d55ccb20e9142a0
/avatars/0e3e80cba64b5ae0bc5638694ac33dbf.svg
Ivan Tang
user
IvanTang
64f552773c35e6e18be109d7
Xianzheng Ma
claimed_verified
2023-09-06T10:49:34.284Z
64f6f80c2d385f3a6d2df467
/avatars/d5ecfafcbbde40cd785f9d6a28f135e7.svg
Xianzheng Ma
user
Oliver-Ma
64f552773c35e6e18be109d8
Jiaming Han
64f552773c35e6e18be109d9
Kexin Chen
64f552773c35e6e18be109da
Peng Gao
64f552773c35e6e18be109db
Xianzhi Li
64f552773c35e6e18be109dc
Hongsheng Li
claimed_verified
2024-02-05T10:50:19.333Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
64f552773c35e6e18be109dd
Pheng-Ann Heng
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D
  Understanding, Generation, and Instruction Following
We introduce Point-Bind, a 3D multi-modality model aligning point clouds with
2D image, language, audio, and video. Guided by ImageBind, we construct a joint
embedding space between 3D and multi-modalities, enabling many promising
applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D
open-world understanding. On top of this, we further present Point-LLM, the
first 3D large language model (LLM) following 3D multi-modal instructions. By
parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of
Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction
data, but exhibits superior 3D and multi-modal question-answering capacity. We
hope our work may cast a light on the community for extending 3D point clouds
to multi-modality applications. Code is available at
https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.
Point-Bind aligns 3D point clouds with various modalities using ImageBind, enabling applications like 3D generation and understanding, while Point-LLM enhances pre-trained LLMs for 3D instructions via parameter-efficient techniques.
Point-Bind 3D point clouds multi-modality ImageBind joint embedding space any-to-3D generation 3D embedding arithmetic 3D open-world understanding Point-LLM 3D large language model parameter-efficient fine-tuning