Scalable 3D Captioning with Pretrained Models
We introduce Cap3D, an automatic approach for generating descriptive text for
3D objects. This approach utilizes pretrained models from image captioning,
image-text alignment, and LLM to consolidate captions from multiple views of a
3D asset, completely side-stepping the time-consuming and costly process of
manual annotation. We apply Cap3D to the recently introduced large-scale 3D
dataset, Objaverse, resulting in 660k 3D-text pairs. Our evaluation, conducted
using 41k human annotations from the same dataset, demonstrates that Cap3D
surpasses human-authored descriptions in terms of quality, cost, and speed.
Through effective prompt engineering, Cap3D rivals human performance in
generating geometric descriptions on 17k collected annotations from the ABO
dataset. Finally, we finetune Text-to-3D models on Cap3D and human captions,
and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shape-E,
and DreamFusion.
6487c5707c22123360e6a4ce
Tiange Luo
admin_assigned
2023-06-13T10:10:19.452Z
61ad3bdb6da53246bd6ac41b
/avatars/19e08cc04c7970db5ff81c043351201b.svg
Tiange
user
tiange
6487c5707c22123360e6a4cf
Chris Rockwell
claimed_verified
2023-06-13T17:39:00.531Z
645297ba1a0c17bb7d5975d6
https://cdn-avatars.huggingface.co/v1/production/uploads/645297ba1a0c17bb7d5975d6/oY7kd0-UcfzYub9ewjAJC.jpeg
Chris Rockwell
user
crockwell
6487c5707c22123360e6a4d0
Honglak Lee
admin_assigned
2023-06-13T10:11:08.896Z
629b37140a85ae646a18018a
/avatars/59192ffb6fe73baf546729e3410c9ecd.svg
Lee
user
Honglak
6487c5707c22123360e6a4d1
Justin Johnson
Scalable 3D Captioning with Pretrained Models
We introduce Cap3D, an automatic approach for generating descriptive text for
3D objects. This approach utilizes pretrained models from image captioning,
image-text alignment, and LLM to consolidate captions from multiple views of a
3D asset, completely side-stepping the time-consuming and costly process of
manual annotation. We apply Cap3D to the recently introduced large-scale 3D
dataset, Objaverse, resulting in 660k 3D-text pairs. Our evaluation, conducted
using 41k human annotations from the same dataset, demonstrates that Cap3D
surpasses human-authored descriptions in terms of quality, cost, and speed.
Through effective prompt engineering, Cap3D rivals human performance in
generating geometric descriptions on 17k collected annotations from the ABO
dataset. Finally, we finetune Text-to-3D models on Cap3D and human captions,
and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shape-E,
and DreamFusion.
Cap3D generates high-quality descriptive text for 3D objects using pretrained models and datasets, surpassing human performance in quality, cost, and speed.
image captioning image-text alignment LLM geometric descriptions prompt engineering Text-to-3D models fine-tuning