title: OpenShape: Scaling Up 3D Shape Representation Towards Open-World
    Understanding
summary: We introduce OpenShape, a method for learning multi-modal joint
  representations of text, image, and point clouds. We adopt the commonly used
  multi-modal contrastive learning framework for representation alignment, but
  with a specific focus on scaling up 3D representations to enable open-world 3D
  shape understanding. To achieve this, we scale up training data by ensembling
  multiple 3D datasets and propose several strategies to automatically filter and
  enrich noisy text descriptions. We also explore and compare strategies for
  scaling 3D backbone networks and introduce a novel hard negative mining module
  for more efficient training. We evaluate OpenShape on zero-shot 3D
  classification benchmarks and demonstrate its superior capabilities for
  open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy
  of 46.8% on the 1,156-category Objaverse-LVIS benchmark, compared to less than
  10% for existing methods. OpenShape also achieves an accuracy of 85.3% on
  ModelNet40, outperforming previous zero-shot baseline methods by 20% and
  performing on par with some fully-supervised methods. Furthermore, we show that
  our learned embeddings encode a wide range of visual and semantic concepts
  (e.g., subcategories, color, shape, style) and facilitate fine-grained text-3D
  and image-3D interactions. Due to their alignment with CLIP embeddings, our
  learned shape representations can also be integrated with off-the-shelf
  CLIP-based models for various applications, such as point cloud captioning and
  point cloud-conditioned image generation.
paper_authors._id: 64670160f3e78d1d6be123cd
paper_authors.name: Minghua Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:04:08.813Z
paper_authors._id: 63b70a696fca9d2a1904f606
paper_authors.avatarUrl: /avatars/a11741787b884cc865c1b7e9e6a7ec24.svg
paper_authors.fullname: Minghua Liu
paper_authors.type: user
paper_authors.user: minghua
paper_authors._id: 64670160f3e78d1d6be123ce
paper_authors.name: Ruoxi Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T07:25:09.334Z
paper_authors._id: 640820798dca6cec91cb2f0a
paper_authors.avatarUrl: /avatars/e011bc04facd251570247d1ae214dbde.svg
paper_authors.fullname: Ruoxi Shi
paper_authors.type: user
paper_authors.user: eliphatfs
paper_authors._id: 64670160f3e78d1d6be123cf
paper_authors.name: Kaiming Kuang
paper_authors._id: 64670160f3e78d1d6be123d0
paper_authors.name: Yinhao Zhu
paper_authors._id: 64670160f3e78d1d6be123d1
paper_authors.name: Xuanlin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-25T08:40:40.639Z
paper_authors._id: 64139dbfb0c7659b8fd54eea
paper_authors.avatarUrl: /avatars/41b552870c7f5159c11135fa4671af35.svg
paper_authors.fullname: Xuanlin Li
paper_authors.type: user
paper_authors.user: xuanlinli17
paper_authors._id: 64670160f3e78d1d6be123d2
paper_authors.name: Shizhong Han
paper_authors._id: 64670160f3e78d1d6be123d3
paper_authors.name: Hong Cai
paper_authors._id: 64670160f3e78d1d6be123d4
paper_authors.name: Fatih Porikli
paper_authors._id: 64670160f3e78d1d6be123d5
paper_authors.name: Hao Su
paper_title: OpenShape: Scaling Up 3D Shape Representation Towards Open-World
    Understanding
paper_summary: We introduce OpenShape, a method for learning multi-modal joint
  representations of text, image, and point clouds. We adopt the commonly used
  multi-modal contrastive learning framework for representation alignment, but
  with a specific focus on scaling up 3D representations to enable open-world 3D
  shape understanding. To achieve this, we scale up training data by ensembling
  multiple 3D datasets and propose several strategies to automatically filter and
  enrich noisy text descriptions. We also explore and compare strategies for
  scaling 3D backbone networks and introduce a novel hard negative mining module
  for more efficient training. We evaluate OpenShape on zero-shot 3D
  classification benchmarks and demonstrate its superior capabilities for
  open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy
  of 46.8% on the 1,156-category Objaverse-LVIS benchmark, compared to less than
  10% for existing methods. OpenShape also achieves an accuracy of 85.3% on
  ModelNet40, outperforming previous zero-shot baseline methods by 20% and
  performing on par with some fully-supervised methods. Furthermore, we show that
  our learned embeddings encode a wide range of visual and semantic concepts
  (e.g., subcategories, color, shape, style) and facilitate fine-grained text-3D
  and image-3D interactions. Due to their alignment with CLIP embeddings, our
  learned shape representations can also be integrated with off-the-shelf
  CLIP-based models for various applications, such as point cloud captioning and
  point cloud-conditioned image generation.
paper_ai_summary: OpenShape learns multi-modal representations for text, images, and 3D point clouds, achieving superior zero-shot performance and fine-grained interactions through novel strategies and hard negative mining.
paper_ai_keywords: multi-modal joint representations multi-modal contrastive learning 3D representations open-world 3D shape understanding 3D datasets noisy text descriptions 3D backbone networks hard negative mining module zero-shot classification Objaverse-LVIS ModelNet40 CLIP embeddings point cloud captioning point cloud-conditioned image generation