title: Real3D: Scaling Up Large Reconstruction Models with Real-World Images
summary: The default strategy for training single-view Large Reconstruction Models
  (LRMs) follows the fully supervised route using large-scale datasets of
  synthetic 3D assets or multi-view captures. Although these resources simplify
  the training procedure, they are hard to scale up beyond the existing datasets
  and they are not necessarily representative of the real distribution of object
  shapes. To address these limitations, in this paper, we introduce Real3D, the
  first LRM system that can be trained using single-view real-world images.
  Real3D introduces a novel self-training framework that can benefit from both
  the existing synthetic data and diverse single-view real images. We propose two
  unsupervised losses that allow us to supervise LRMs at the pixel- and
  semantic-level, even for training examples without ground-truth 3D or novel
  views. To further improve performance and scale up the image data, we develop
  an automatic data curation approach to collect high-quality examples from
  in-the-wild images. Our experiments show that Real3D consistently outperforms
  prior work in four diverse evaluation settings that include real and synthetic
  data, as well as both in-domain and out-of-domain shapes. Code and model can be
  found here: https://hwjiang1510.github.io/Real3D/
paper_authors._id: 666b0dd1a10c1e31ff37e704
paper_authors.name: Hanwen Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:26:03.683Z
paper_authors._id: 6666692aca15c6233f414ac2
paper_authors.avatarUrl: /avatars/462841d3ba16d351182e9aa552c36044.svg
paper_authors.fullname: Hanwen Jiang
paper_authors.type: user
paper_authors.user: hwjiang
paper_authors._id: 666b0dd1a10c1e31ff37e705
paper_authors.name: Qixing Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T07:01:28.638Z
paper_authors._id: 67f18d051358145ee5bb3a11
paper_authors.avatarUrl: /avatars/e5a8c12295d7609625c2b891d47c3112.svg
paper_authors.fullname: qixing huang
paper_authors.type: user
paper_authors.user: qixinghuang1
paper_authors._id: 666b0dd1a10c1e31ff37e706
paper_authors.name: Georgios Pavlakos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:26:29.655Z
paper_authors._id: 6478d3433b7f8b1f6249b469
paper_authors.avatarUrl: /avatars/11e7d7a94ae26500c1c2ad62e760726f.svg
paper_authors.fullname: Georgios Pavlakos
paper_authors.type: user
paper_authors.user: geopavlakos
paper_title: Real3D: Scaling Up Large Reconstruction Models with Real-World Images
paper_summary: The default strategy for training single-view Large Reconstruction Models
  (LRMs) follows the fully supervised route using large-scale datasets of
  synthetic 3D assets or multi-view captures. Although these resources simplify
  the training procedure, they are hard to scale up beyond the existing datasets
  and they are not necessarily representative of the real distribution of object
  shapes. To address these limitations, in this paper, we introduce Real3D, the
  first LRM system that can be trained using single-view real-world images.
  Real3D introduces a novel self-training framework that can benefit from both
  the existing synthetic data and diverse single-view real images. We propose two
  unsupervised losses that allow us to supervise LRMs at the pixel- and
  semantic-level, even for training examples without ground-truth 3D or novel
  views. To further improve performance and scale up the image data, we develop
  an automatic data curation approach to collect high-quality examples from
  in-the-wild images. Our experiments show that Real3D consistently outperforms
  prior work in four diverse evaluation settings that include real and synthetic
  data, as well as both in-domain and out-of-domain shapes. Code and model can be
  found here: https://hwjiang1510.github.io/Real3D/
paper_ai_summary: Real3D is an unsupervised LRM training framework that leverages both synthetic and real single-view images to improve performance across various evaluation settings.
paper_ai_keywords: Large Reconstruction Models LRM Real3D self-training framework unsupervised losses pixel-level semantic-level automatic data curation in-the-wild images