ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding
Recent advancements in multimodal pre-training methods have shown promising
efficacy in 3D representation learning by aligning features across 3D modality,
their 2D counterpart modality, and corresponding language modality. However,
the methods used by existing multimodal pre-training frameworks to gather
multimodal data for 3D applications lack scalability and comprehensiveness,
potentially constraining the full potential of multimodal learning. The main
bottleneck lies in the language modality's scalability and comprehensiveness.
To address this bottleneck, we introduce ULIP-2, a multimodal pre-training
framework that leverages state-of-the-art multimodal large language models
(LLMs) pre-trained on extensive knowledge to automatically generate holistic
language counterparts for 3D objects. We conduct experiments on two large-scale
datasets, Objaverse and ShapeNet55, and release our generated three-modality
triplet datasets (3D Point Cloud - Image - Language), named "ULIP-Objaverse
Triplets" and "ULIP-ShapeNet Triplets". ULIP-2 requires only 3D data itself and
eliminates the need for any manual annotation effort, demonstrating its
scalability; and ULIP-2 achieves remarkable improvements on downstream
zero-shot classification on ModelNet40 (74% Top1 Accuracy). Moreover, ULIP-2
sets a new record on the real-world ScanObjectNN benchmark (91.5% Overall
Accuracy) while utilizing only 1.4 million parameters(~10x fewer than current
SOTA), signifying a breakthrough in scalable multimodal 3D representation
learning without human annotations. The code and datasets are available at
https://github.com/salesforce/ULIP.
6462ee412538819c72a0c637
Le Xue
extracted_confirmed
2023-05-16T05:01:38.218Z
63dd73e7422ca8d7f7e3698c
/avatars/7b0f8419f6941230b81dbbbb4f273edf.svg
Le Xue
user
SFXX
6462ee412538819c72a0c638
Ning Yu
claimed_verified
2023-05-17T08:04:47.780Z
6362bcbe8f43a912fc722969
https://cdn-avatars.huggingface.co/v1/production/uploads/6362bcbe8f43a912fc722969/ktl2ePfpOIseqlIbuldNa.png
Ning Yu
user
ningyu1991
6462ee412538819c72a0c639
Shu Zhang
6462ee412538819c72a0c63a
Junnan Li
claimed_verified
2023-05-16T12:25:11.990Z
61f9d3b54ac99e8a1bae85f4
/avatars/ac47d13204dd22452e4bc46e280842d5.svg
JunnanLi
user
JunnanLi
6462ee412538819c72a0c63b
Roberto Martín-Martín
claimed_verified
2023-05-24T07:51:13.771Z
6463d141e0475a83333c240a
/avatars/c0c483bc379c660c34cb8b9c6d491129.svg
Roberto Martin-Martin
user
robertomm
6462ee412538819c72a0c63c
Jiajun Wu
claimed_verified
2023-06-05T10:07:08.531Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
6462ee412538819c72a0c63d
Caiming Xiong
6462ee412538819c72a0c63e
Ran Xu
claimed_verified
2023-05-19T10:27:59.457Z
6465c4c863e7e09dd02e3e1b
/avatars/200b029184d2616f98296a2c212f0785.svg
Ran Xu
user
xurantju
6462ee412538819c72a0c63f
Juan Carlos Niebles
claimed_verified
2024-02-01T08:45:21.919Z
65bb26b5d61b51a50805aba2
/avatars/966c8469f55776948a7cf42d98accff3.svg
Juan Carlos Niebles
user
niebles
6462ee412538819c72a0c640
Silvio Savarese
ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding
Recent advancements in multimodal pre-training methods have shown promising
efficacy in 3D representation learning by aligning features across 3D modality,
their 2D counterpart modality, and corresponding language modality. However,
the methods used by existing multimodal pre-training frameworks to gather
multimodal data for 3D applications lack scalability and comprehensiveness,
potentially constraining the full potential of multimodal learning. The main
bottleneck lies in the language modality's scalability and comprehensiveness.
To address this bottleneck, we introduce ULIP-2, a multimodal pre-training
framework that leverages state-of-the-art multimodal large language models
(LLMs) pre-trained on extensive knowledge to automatically generate holistic
language counterparts for 3D objects. We conduct experiments on two large-scale
datasets, Objaverse and ShapeNet55, and release our generated three-modality
triplet datasets (3D Point Cloud - Image - Language), named "ULIP-Objaverse
Triplets" and "ULIP-ShapeNet Triplets". ULIP-2 requires only 3D data itself and
eliminates the need for any manual annotation effort, demonstrating its
scalability; and ULIP-2 achieves remarkable improvements on downstream
zero-shot classification on ModelNet40 (74% Top1 Accuracy). Moreover, ULIP-2
sets a new record on the real-world ScanObjectNN benchmark (91.5% Overall
Accuracy) while utilizing only 1.4 million parameters(~10x fewer than current
SOTA), signifying a breakthrough in scalable multimodal 3D representation
learning without human annotations. The code and datasets are available at
https://github.com/salesforce/ULIP.
ULIP-2 generates language descriptions for 3D objects using large language models, enabling scalable multimodal 3D representation learning with improved accuracy and fewer parameters.
multimodal pre-training 3D representation learning language modality state-of-the-art multimodal large language models 3D Point Cloud zero-shot classification ScanObjectNN benchmark parameter-efficient