title: Latent Radiance Fields with 3D-aware 2D Representations
summary: Latent 3D reconstruction has shown great promise in empowering 3D semantic
  understanding and 3D generation by distilling 2D features into the 3D space.
  However, existing approaches struggle with the domain gap between 2D feature
  space and 3D representations, resulting in degraded rendering performance. To
  address this challenge, we propose a novel framework that integrates 3D
  awareness into the 2D latent space. The framework consists of three stages: (1)
  a correspondence-aware autoencoding method that enhances the 3D consistency of
  2D latent representations, (2) a latent radiance field (LRF) that lifts these
  3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field
  (VAE-RF) alignment strategy that improves image decoding from the rendered 2D
  representations. Extensive experiments demonstrate that our method outperforms
  the state-of-the-art latent 3D reconstruction approaches in terms of synthesis
  performance and cross-dataset generalizability across diverse indoor and
  outdoor scenes. To our knowledge, this is the first work showing the radiance
  field representations constructed from 2D latent representations can yield
  photorealistic 3D reconstruction performance.
paper_authors._id: 67aff152910c82946ece0343
paper_authors.name: Chaoyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:33:45.516Z
paper_authors._id: 65495d1008775ce78e43e77d
paper_authors.avatarUrl: /avatars/172678e8187acfec0aa8e647479cbb81.svg
paper_authors.fullname: Chaoyi Zhou
paper_authors.type: user
paper_authors.user: chaoyizh
paper_authors._id: 67aff152910c82946ece0344
paper_authors.name: Xi Liu
paper_authors._id: 67aff152910c82946ece0345
paper_authors.name: Feng Luo
paper_authors._id: 67aff152910c82946ece0346
paper_authors.name: Siyu Huang
paper_title: Latent Radiance Fields with 3D-aware 2D Representations
paper_summary: Latent 3D reconstruction has shown great promise in empowering 3D semantic
  understanding and 3D generation by distilling 2D features into the 3D space.
  However, existing approaches struggle with the domain gap between 2D feature
  space and 3D representations, resulting in degraded rendering performance. To
  address this challenge, we propose a novel framework that integrates 3D
  awareness into the 2D latent space. The framework consists of three stages: (1)
  a correspondence-aware autoencoding method that enhances the 3D consistency of
  2D latent representations, (2) a latent radiance field (LRF) that lifts these
  3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field
  (VAE-RF) alignment strategy that improves image decoding from the rendered 2D
  representations. Extensive experiments demonstrate that our method outperforms
  the state-of-the-art latent 3D reconstruction approaches in terms of synthesis
  performance and cross-dataset generalizability across diverse indoor and
  outdoor scenes. To our knowledge, this is the first work showing the radiance
  field representations constructed from 2D latent representations can yield
  photorealistic 3D reconstruction performance.
paper_ai_summary: A novel framework integrates 3D awareness into 2D latent space for enhanced 3D reconstruction, achieving state-of-the-art performance and photorealism across diverse scenes.
paper_ai_keywords: latent 3D reconstruction 3D semantic understanding 3D generation domain gap 2D feature space 3D representations correspondence-aware autoencoding latent radiance field VAE-Radiance Field VAE-RF alignment strategy