title: Scalable 3D Captioning with Pretrained Models
summary: We introduce Cap3D, an automatic approach for generating descriptive text for
  3D objects. This approach utilizes pretrained models from image captioning,
  image-text alignment, and LLM to consolidate captions from multiple views of a
  3D asset, completely side-stepping the time-consuming and costly process of
  manual annotation. We apply Cap3D to the recently introduced large-scale 3D
  dataset, Objaverse, resulting in 660k 3D-text pairs. Our evaluation, conducted
  using 41k human annotations from the same dataset, demonstrates that Cap3D
  surpasses human-authored descriptions in terms of quality, cost, and speed.
  Through effective prompt engineering, Cap3D rivals human performance in
  generating geometric descriptions on 17k collected annotations from the ABO
  dataset. Finally, we finetune Text-to-3D models on Cap3D and human captions,
  and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shape-E,
  and DreamFusion.
paper_authors._id: 6487c5707c22123360e6a4ce
paper_authors.name: Tiange Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T10:10:19.452Z
paper_authors._id: 61ad3bdb6da53246bd6ac41b
paper_authors.avatarUrl: /avatars/19e08cc04c7970db5ff81c043351201b.svg
paper_authors.fullname: Tiange
paper_authors.type: user
paper_authors.user: tiange
paper_authors._id: 6487c5707c22123360e6a4cf
paper_authors.name: Chris Rockwell
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-13T17:39:00.531Z
paper_authors._id: 645297ba1a0c17bb7d5975d6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645297ba1a0c17bb7d5975d6/LA7UInXyxhCsdW412arIc.png
paper_authors.fullname: Chris Rockwell
paper_authors.type: user
paper_authors.user: crockwell
paper_authors._id: 6487c5707c22123360e6a4d0
paper_authors.name: Honglak Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T10:11:08.896Z
paper_authors._id: 629b37140a85ae646a18018a
paper_authors.avatarUrl: /avatars/59192ffb6fe73baf546729e3410c9ecd.svg
paper_authors.fullname: Lee
paper_authors.type: user
paper_authors.user: Honglak
paper_authors._id: 6487c5707c22123360e6a4d1
paper_authors.name: Justin Johnson
paper_title: Scalable 3D Captioning with Pretrained Models
paper_summary: We introduce Cap3D, an automatic approach for generating descriptive text for
  3D objects. This approach utilizes pretrained models from image captioning,
  image-text alignment, and LLM to consolidate captions from multiple views of a
  3D asset, completely side-stepping the time-consuming and costly process of
  manual annotation. We apply Cap3D to the recently introduced large-scale 3D
  dataset, Objaverse, resulting in 660k 3D-text pairs. Our evaluation, conducted
  using 41k human annotations from the same dataset, demonstrates that Cap3D
  surpasses human-authored descriptions in terms of quality, cost, and speed.
  Through effective prompt engineering, Cap3D rivals human performance in
  generating geometric descriptions on 17k collected annotations from the ABO
  dataset. Finally, we finetune Text-to-3D models on Cap3D and human captions,
  and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shape-E,
  and DreamFusion.
paper_ai_keywords: Cap3D image captioning image-text alignment LLM Objaverse 3D-text pairs human annotations prompt engineering ABO dataset Text-to-3D models Point-E Shape-E DreamFusion