OpenShape: Scaling Up 3D Shape Representation Towards Open-World
  Understanding
We introduce OpenShape, a method for learning multi-modal joint
representations of text, image, and point clouds. We adopt the commonly used
multi-modal contrastive learning framework for representation alignment, but
with a specific focus on scaling up 3D representations to enable open-world 3D
shape understanding. To achieve this, we scale up training data by ensembling
multiple 3D datasets and propose several strategies to automatically filter and
enrich noisy text descriptions. We also explore and compare strategies for
scaling 3D backbone networks and introduce a novel hard negative mining module
for more efficient training. We evaluate OpenShape on zero-shot 3D
classification benchmarks and demonstrate its superior capabilities for
open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy
of 46.8% on the 1,156-category Objaverse-LVIS benchmark, compared to less than
10% for existing methods. OpenShape also achieves an accuracy of 85.3% on
ModelNet40, outperforming previous zero-shot baseline methods by 20% and
performing on par with some fully-supervised methods. Furthermore, we show that
our learned embeddings encode a wide range of visual and semantic concepts
(e.g., subcategories, color, shape, style) and facilitate fine-grained text-3D
and image-3D interactions. Due to their alignment with CLIP embeddings, our
learned shape representations can also be integrated with off-the-shelf
CLIP-based models for various applications, such as point cloud captioning and
point cloud-conditioned image generation.
64670160f3e78d1d6be123cd
Minghua Liu
claimed_verified
2023-05-19T19:04:08.813Z
63b70a696fca9d2a1904f606
/avatars/a11741787b884cc865c1b7e9e6a7ec24.svg
Minghua Liu
user
minghua
64670160f3e78d1d6be123ce
Ruoxi Shi
admin_assigned
2023-06-14T07:25:09.334Z
640820798dca6cec91cb2f0a
/avatars/e011bc04facd251570247d1ae214dbde.svg
Ruoxi Shi
user
eliphatfs
64670160f3e78d1d6be123cf
Kaiming Kuang
64670160f3e78d1d6be123d0
Yinhao Zhu
64670160f3e78d1d6be123d1
Xuanlin Li
claimed_verified
2023-05-25T08:40:40.639Z
64139dbfb0c7659b8fd54eea
/avatars/41b552870c7f5159c11135fa4671af35.svg
Xuanlin Li
user
xuanlinli17
64670160f3e78d1d6be123d2
Shizhong Han
64670160f3e78d1d6be123d3
Hong Cai
64670160f3e78d1d6be123d4
Fatih Porikli
64670160f3e78d1d6be123d5
Hao Su
OpenShape: Scaling Up 3D Shape Representation Towards Open-World
  Understanding
We introduce OpenShape, a method for learning multi-modal joint
representations of text, image, and point clouds. We adopt the commonly used
multi-modal contrastive learning framework for representation alignment, but
with a specific focus on scaling up 3D representations to enable open-world 3D
shape understanding. To achieve this, we scale up training data by ensembling
multiple 3D datasets and propose several strategies to automatically filter and
enrich noisy text descriptions. We also explore and compare strategies for
scaling 3D backbone networks and introduce a novel hard negative mining module
for more efficient training. We evaluate OpenShape on zero-shot 3D
classification benchmarks and demonstrate its superior capabilities for
open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy
of 46.8% on the 1,156-category Objaverse-LVIS benchmark, compared to less than
10% for existing methods. OpenShape also achieves an accuracy of 85.3% on
ModelNet40, outperforming previous zero-shot baseline methods by 20% and
performing on par with some fully-supervised methods. Furthermore, we show that
our learned embeddings encode a wide range of visual and semantic concepts
(e.g., subcategories, color, shape, style) and facilitate fine-grained text-3D
and image-3D interactions. Due to their alignment with CLIP embeddings, our
learned shape representations can also be integrated with off-the-shelf
CLIP-based models for various applications, such as point cloud captioning and
point cloud-conditioned image generation.
OpenShape learns multi-modal representations for text, images, and 3D point clouds, achieving superior zero-shot performance and fine-grained interactions through novel strategies and hard negative mining.
multi-modal joint representations multi-modal contrastive learning 3D representations open-world 3D shape understanding 3D datasets noisy text descriptions 3D backbone networks hard negative mining module zero-shot classification Objaverse-LVIS ModelNet40 CLIP embeddings point cloud captioning point cloud-conditioned image generation