title: GTR: Improving Large 3D Reconstruction Models through Geometry and
    Texture Refinement
summary: We propose a novel approach for 3D mesh reconstruction from multi-view
  images. Our method takes inspiration from large reconstruction models like LRM
  that use a transformer-based triplane generator and a Neural Radiance Field
  (NeRF) model trained on multi-view images. However, in our method, we introduce
  several important modifications that allow us to significantly enhance 3D
  reconstruction quality. First of all, we examine the original LRM architecture
  and find several shortcomings. Subsequently, we introduce respective
  modifications to the LRM architecture, which lead to improved multi-view image
  representation and more computationally efficient training. Second, in order to
  improve geometry reconstruction and enable supervision at full image
  resolution, we extract meshes from the NeRF field in a differentiable manner
  and fine-tune the NeRF model through mesh rendering. These modifications allow
  us to achieve state-of-the-art performance on both 2D and 3D evaluation
  metrics, such as a PSNR of 28.67 on Google Scanned Objects (GSO) dataset.
  Despite these superior results, our feed-forward model still struggles to
  reconstruct complex textures, such as text and portraits on assets. To address
  this, we introduce a lightweight per-instance texture refinement procedure.
  This procedure fine-tunes the triplane representation and the NeRF color
  estimation model on the mesh surface using the input multi-view images in just
  4 seconds. This refinement improves the PSNR to 29.79 and achieves faithful
  reconstruction of complex textures, such as text. Additionally, our approach
  enables various downstream applications, including text- or image-to-3D
  generation.
paper_authors._id: 6667d118d777e82ef9f0939f
paper_authors.name: Peiye Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:14:30.677Z
paper_authors._id: 63e473cf9db5da2dc1f4e9ff
paper_authors.avatarUrl: /avatars/6973b4c959ef8bb72c40c611d384a78a.svg
paper_authors.fullname: peiye zhuang
paper_authors.type: user
paper_authors.user: Kelest
paper_authors._id: 6667d118d777e82ef9f093a0
paper_authors.name: Songfang Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:14:40.440Z
paper_authors._id: 63085d8ef0dc38fb47c3aa6b
paper_authors.avatarUrl: /avatars/83bdc8a4f8a2ca78a854a67cb634b3ef.svg
paper_authors.fullname: Han
paper_authors.type: user
paper_authors.user: Songfang
paper_authors._id: 6667d118d777e82ef9f093a1
paper_authors.name: Chaoyang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:15:02.964Z
paper_authors._id: 64c166de11f3f1d23a99c6fd
paper_authors.avatarUrl: /avatars/d4730f70a6f1cb51266862c7a8d54f77.svg
paper_authors.fullname: Chaoyang Wang
paper_authors.type: user
paper_authors.user: cwang9
paper_authors._id: 6667d118d777e82ef9f093a2
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:15:09.377Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 6667d118d777e82ef9f093a3
paper_authors.name: Jiaxu Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:15:18.617Z
paper_authors._id: 643ac5720e5495afdefa1a51
paper_authors.avatarUrl: /avatars/37fdbdca6e683f98cbbc6c048a9b86bc.svg
paper_authors.fullname: Jiaxu Zou
paper_authors.type: user
paper_authors.user: zoujx96
paper_authors._id: 6667d118d777e82ef9f093a4
paper_authors.name: Michael Vasilkovsky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:15:24.519Z
paper_authors._id: 64775c34dbc2a416f8b648f6
paper_authors.avatarUrl: /avatars/014b7011e0bd2e8ab6fa91a71ddb2d60.svg
paper_authors.fullname: Michael Vasilkovsky
paper_authors.type: user
paper_authors.user: waytobehigh
paper_authors._id: 6667d118d777e82ef9f093a5
paper_authors.name: Vladislav Shakhrai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:15:30.502Z
paper_authors._id: 640290115caf6d21d67ea5f3
paper_authors.avatarUrl: /avatars/236fcd4ab50e6305ca0f76317731a63c.svg
paper_authors.fullname: Vladislav Shakhrai
paper_authors.type: user
paper_authors.user: shakhrayv
paper_authors._id: 6667d118d777e82ef9f093a6
paper_authors.name: Sergey Korolev
paper_authors._id: 6667d118d777e82ef9f093a7
paper_authors.name: Sergey Tulyakov
paper_authors._id: 6667d118d777e82ef9f093a8
paper_authors.name: Hsin-Ying Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:15:53.329Z
paper_authors._id: 631bbd918de8e645af71eed9
paper_authors.avatarUrl: /avatars/1cb94deadc288d07c571c9289085548c.svg
paper_authors.fullname: Hsin-Ying Lee
paper_authors.type: user
paper_authors.user: james371507
paper_title: GTR: Improving Large 3D Reconstruction Models through Geometry and
    Texture Refinement
paper_summary: We propose a novel approach for 3D mesh reconstruction from multi-view
  images. Our method takes inspiration from large reconstruction models like LRM
  that use a transformer-based triplane generator and a Neural Radiance Field
  (NeRF) model trained on multi-view images. However, in our method, we introduce
  several important modifications that allow us to significantly enhance 3D
  reconstruction quality. First of all, we examine the original LRM architecture
  and find several shortcomings. Subsequently, we introduce respective
  modifications to the LRM architecture, which lead to improved multi-view image
  representation and more computationally efficient training. Second, in order to
  improve geometry reconstruction and enable supervision at full image
  resolution, we extract meshes from the NeRF field in a differentiable manner
  and fine-tune the NeRF model through mesh rendering. These modifications allow
  us to achieve state-of-the-art performance on both 2D and 3D evaluation
  metrics, such as a PSNR of 28.67 on Google Scanned Objects (GSO) dataset.
  Despite these superior results, our feed-forward model still struggles to
  reconstruct complex textures, such as text and portraits on assets. To address
  this, we introduce a lightweight per-instance texture refinement procedure.
  This procedure fine-tunes the triplane representation and the NeRF color
  estimation model on the mesh surface using the input multi-view images in just
  4 seconds. This refinement improves the PSNR to 29.79 and achieves faithful
  reconstruction of complex textures, such as text. Additionally, our approach
  enables various downstream applications, including text- or image-to-3D
  generation.
paper_ai_keywords: transformer-based triplane generator Neural Radiance Field (NeRF) multi-view images LRM architecture 3D reconstruction differentiable manner mesh rendering PSNR Google Scanned Objects (GSO) dataset texture refinement procedure triplane representation NeRF color estimation model text- or image-to-3D generation