title: ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding
summary: Recent advancements in multimodal pre-training methods have shown promising
  efficacy in 3D representation learning by aligning features across 3D modality,
  their 2D counterpart modality, and corresponding language modality. However,
  the methods used by existing multimodal pre-training frameworks to gather
  multimodal data for 3D applications lack scalability and comprehensiveness,
  potentially constraining the full potential of multimodal learning. The main
  bottleneck lies in the language modality's scalability and comprehensiveness.
  To address this bottleneck, we introduce ULIP-2, a multimodal pre-training
  framework that leverages state-of-the-art multimodal large language models
  (LLMs) pre-trained on extensive knowledge to automatically generate holistic
  language counterparts for 3D objects. We conduct experiments on two large-scale
  datasets, Objaverse and ShapeNet55, and release our generated three-modality
  triplet datasets (3D Point Cloud - Image - Language), named "ULIP-Objaverse
  Triplets" and "ULIP-ShapeNet Triplets". ULIP-2 requires only 3D data itself and
  eliminates the need for any manual annotation effort, demonstrating its
  scalability; and ULIP-2 achieves remarkable improvements on downstream
  zero-shot classification on ModelNet40 (74% Top1 Accuracy). Moreover, ULIP-2
  sets a new record on the real-world ScanObjectNN benchmark (91.5% Overall
  Accuracy) while utilizing only 1.4 million parameters(~10x fewer than current
  SOTA), signifying a breakthrough in scalable multimodal 3D representation
  learning without human annotations. The code and datasets are available at
  https://github.com/salesforce/ULIP.
paper_authors._id: 6462ee412538819c72a0c637
paper_authors.name: Le Xue
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-16T05:01:38.218Z
paper_authors._id: 63dd73e7422ca8d7f7e3698c
paper_authors.avatarUrl: /avatars/7b0f8419f6941230b81dbbbb4f273edf.svg
paper_authors.fullname: Le Xue
paper_authors.type: user
paper_authors.user: SFXX
paper_authors._id: 6462ee412538819c72a0c638
paper_authors.name: Ning Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-17T08:04:47.780Z
paper_authors._id: 6362bcbe8f43a912fc722969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6362bcbe8f43a912fc722969/ktl2ePfpOIseqlIbuldNa.png
paper_authors.fullname: Ning Yu
paper_authors.type: user
paper_authors.user: ningyu1991
paper_authors._id: 6462ee412538819c72a0c639
paper_authors.name: Shu Zhang
paper_authors._id: 6462ee412538819c72a0c63a
paper_authors.name: Junnan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T12:25:11.990Z
paper_authors._id: 61f9d3b54ac99e8a1bae85f4
paper_authors.avatarUrl: /avatars/ac47d13204dd22452e4bc46e280842d5.svg
paper_authors.fullname: JunnanLi
paper_authors.type: user
paper_authors.user: JunnanLi
paper_authors._id: 6462ee412538819c72a0c63b
paper_authors.name: Roberto Martín-Martín
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-24T07:51:13.771Z
paper_authors._id: 6463d141e0475a83333c240a
paper_authors.avatarUrl: /avatars/c0c483bc379c660c34cb8b9c6d491129.svg
paper_authors.fullname: Roberto Martin-Martin
paper_authors.type: user
paper_authors.user: robertomm
paper_authors._id: 6462ee412538819c72a0c63c
paper_authors.name: Jiajun Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:07:08.531Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_authors._id: 6462ee412538819c72a0c63d
paper_authors.name: Caiming Xiong
paper_authors._id: 6462ee412538819c72a0c63e
paper_authors.name: Ran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T10:27:59.457Z
paper_authors._id: 6465c4c863e7e09dd02e3e1b
paper_authors.avatarUrl: /avatars/200b029184d2616f98296a2c212f0785.svg
paper_authors.fullname: Ran Xu
paper_authors.type: user
paper_authors.user: xurantju
paper_authors._id: 6462ee412538819c72a0c63f
paper_authors.name: Juan Carlos Niebles
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-01T08:45:21.919Z
paper_authors._id: 65bb26b5d61b51a50805aba2
paper_authors.avatarUrl: /avatars/966c8469f55776948a7cf42d98accff3.svg
paper_authors.fullname: Juan Carlos Niebles
paper_authors.type: user
paper_authors.user: niebles
paper_authors._id: 6462ee412538819c72a0c640
paper_authors.name: Silvio Savarese
paper_title: ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding
paper_summary: Recent advancements in multimodal pre-training methods have shown promising
  efficacy in 3D representation learning by aligning features across 3D modality,
  their 2D counterpart modality, and corresponding language modality. However,
  the methods used by existing multimodal pre-training frameworks to gather
  multimodal data for 3D applications lack scalability and comprehensiveness,
  potentially constraining the full potential of multimodal learning. The main
  bottleneck lies in the language modality's scalability and comprehensiveness.
  To address this bottleneck, we introduce ULIP-2, a multimodal pre-training
  framework that leverages state-of-the-art multimodal large language models
  (LLMs) pre-trained on extensive knowledge to automatically generate holistic
  language counterparts for 3D objects. We conduct experiments on two large-scale
  datasets, Objaverse and ShapeNet55, and release our generated three-modality
  triplet datasets (3D Point Cloud - Image - Language), named "ULIP-Objaverse
  Triplets" and "ULIP-ShapeNet Triplets". ULIP-2 requires only 3D data itself and
  eliminates the need for any manual annotation effort, demonstrating its
  scalability; and ULIP-2 achieves remarkable improvements on downstream
  zero-shot classification on ModelNet40 (74% Top1 Accuracy). Moreover, ULIP-2
  sets a new record on the real-world ScanObjectNN benchmark (91.5% Overall
  Accuracy) while utilizing only 1.4 million parameters(~10x fewer than current
  SOTA), signifying a breakthrough in scalable multimodal 3D representation
  learning without human annotations. The code and datasets are available at
  https://github.com/salesforce/ULIP.
paper_ai_keywords: multimodal pre-training methods 3D representation learning multimodal large language models (LLMs) holistic language counterparts 3D Point Cloud Image Language zero-shot demonstration ModelNet40 ScanObjectNN 3D data manual annotation scalable multimodal 3D representation learning