GPT4Point: A Unified Framework for Point-Language Understanding and
  Generation
Multimodal Large Language Models (MLLMs) have excelled in 2D image-text
comprehension and image generation, but their understanding of the 3D world is
notably deficient, limiting progress in 3D language understanding and
generation. To solve this problem, we introduce GPT4Point, an innovative
groundbreaking point-language multimodal model designed specifically for
unified 3D object understanding and generation within the MLLM framework.
GPT4Point as a powerful 3D MLLM seamlessly can execute a variety of point-text
reference tasks such as point-cloud captioning and Q&A. Additionally, GPT4Point
is equipped with advanced capabilities for controllable 3D generation, it can
get high-quality results through a low-quality point-text feature maintaining
the geometric shapes and colors. To support the expansive needs of 3D
object-text pairs, we develop Pyramid-XL, a point-language dataset annotation
engine. It constructs a large-scale database over 1M objects of varied text
granularity levels from the Objaverse-XL dataset, essential for training
GPT4Point. A comprehensive benchmark has been proposed to evaluate 3D
point-language understanding capabilities. In extensive evaluations, GPT4Point
has demonstrated superior performance in understanding and generation.
656fd84580ff527c44fac4f7
Zhangyang Qi
admin_assigned
2023-12-06T12:55:47.435Z
64c129c155686a357c99ea0b
https://cdn-avatars.huggingface.co/v1/production/uploads/64c129c155686a357c99ea0b/1iy8_aLVU4FflO-d1fozO.jpeg
Qi Zhangyang (Alex Severns)
user
alexzyqi
656fd84580ff527c44fac4f8
Ye Fang
claimed_verified
2023-12-08T12:08:21.681Z
6564c4aacb47db1a9b8f72a2
https://cdn-avatars.huggingface.co/v1/production/uploads/6564c4aacb47db1a9b8f72a2/Mgv25wB1sea3d76vigbx8.jpeg
Ye Fang
user
aleafy
656fd84580ff527c44fac4f9
Zeyi Sun
admin_assigned
2023-12-06T12:56:26.559Z
63fda3fced9eead590ff6918
https://cdn-avatars.huggingface.co/v1/production/uploads/1677566802735-noauth.jpeg
Zeyi Sun
user
Zery
656fd84580ff527c44fac4fa
Xiaoyang Wu
admin_assigned
2023-12-06T12:56:54.042Z
643e5d6a1d0e956d94bb3608
https://cdn-avatars.huggingface.co/v1/production/uploads/643e5d6a1d0e956d94bb3608/mTnu7Dts2RmDklHlp9Gqu.jpeg
Xiaoyang Wu
user
Gofinge
656fd84580ff527c44fac4fb
Tong Wu
claimed_verified
2024-12-11T09:25:06.220Z
6310929fda0aaad2f70397a1
https://cdn-avatars.huggingface.co/v1/production/uploads/6310929fda0aaad2f70397a1/YoHeYZezmCOXwL-qsiZOc.png
Tong Wu
user
wutong16
656fd84580ff527c44fac4fc
Jiaqi Wang
claimed_verified
2024-06-09T20:07:29.314Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
656fd84580ff527c44fac4fd
Dahua Lin
admin_assigned
2023-12-06T13:00:04.504Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
656fd84580ff527c44fac4fe
Hengshuang Zhao
GPT4Point: A Unified Framework for Point-Language Understanding and
  Generation
Multimodal Large Language Models (MLLMs) have excelled in 2D image-text
comprehension and image generation, but their understanding of the 3D world is
notably deficient, limiting progress in 3D language understanding and
generation. To solve this problem, we introduce GPT4Point, an innovative
groundbreaking point-language multimodal model designed specifically for
unified 3D object understanding and generation within the MLLM framework.
GPT4Point as a powerful 3D MLLM seamlessly can execute a variety of point-text
reference tasks such as point-cloud captioning and Q&A. Additionally, GPT4Point
is equipped with advanced capabilities for controllable 3D generation, it can
get high-quality results through a low-quality point-text feature maintaining
the geometric shapes and colors. To support the expansive needs of 3D
object-text pairs, we develop Pyramid-XL, a point-language dataset annotation
engine. It constructs a large-scale database over 1M objects of varied text
granularity levels from the Objaverse-XL dataset, essential for training
GPT4Point. A comprehensive benchmark has been proposed to evaluate 3D
point-language understanding capabilities. In extensive evaluations, GPT4Point
has demonstrated superior performance in understanding and generation.
GPT4Point, a 3D point-language multimodal model, enables effective 3D object understanding and generation with advanced capabilities for point-cloud captioning and controllable generation.
GPT4Point 3D MLLM point-cloud captioning point-cloud Q&A controllable 3D generation Pyramid-XL point-language dataset annotation engine Objaverse-XL comprehensive benchmark