GTR: Improving Large 3D Reconstruction Models through Geometry and
  Texture Refinement
We propose a novel approach for 3D mesh reconstruction from multi-view
images. Our method takes inspiration from large reconstruction models like LRM
that use a transformer-based triplane generator and a Neural Radiance Field
(NeRF) model trained on multi-view images. However, in our method, we introduce
several important modifications that allow us to significantly enhance 3D
reconstruction quality. First of all, we examine the original LRM architecture
and find several shortcomings. Subsequently, we introduce respective
modifications to the LRM architecture, which lead to improved multi-view image
representation and more computationally efficient training. Second, in order to
improve geometry reconstruction and enable supervision at full image
resolution, we extract meshes from the NeRF field in a differentiable manner
and fine-tune the NeRF model through mesh rendering. These modifications allow
us to achieve state-of-the-art performance on both 2D and 3D evaluation
metrics, such as a PSNR of 28.67 on Google Scanned Objects (GSO) dataset.
Despite these superior results, our feed-forward model still struggles to
reconstruct complex textures, such as text and portraits on assets. To address
this, we introduce a lightweight per-instance texture refinement procedure.
This procedure fine-tunes the triplane representation and the NeRF color
estimation model on the mesh surface using the input multi-view images in just
4 seconds. This refinement improves the PSNR to 29.79 and achieves faithful
reconstruction of complex textures, such as text. Additionally, our approach
enables various downstream applications, including text- or image-to-3D
generation.
6667d118d777e82ef9f0939f
Peiye Zhuang
admin_assigned
2024-06-11T10:14:30.677Z
63e473cf9db5da2dc1f4e9ff
/avatars/6973b4c959ef8bb72c40c611d384a78a.svg
peiye zhuang
user
Kelest
6667d118d777e82ef9f093a0
Songfang Han
admin_assigned
2024-06-11T10:14:40.440Z
63085d8ef0dc38fb47c3aa6b
/avatars/83bdc8a4f8a2ca78a854a67cb634b3ef.svg
Han
user
Songfang
6667d118d777e82ef9f093a1
Chaoyang Wang
admin_assigned
2024-06-11T10:15:02.964Z
64c166de11f3f1d23a99c6fd
/avatars/d4730f70a6f1cb51266862c7a8d54f77.svg
Chaoyang Wang
user
cwang9
6667d118d777e82ef9f093a2
Aliaksandr Siarohin
admin_assigned
2024-06-11T10:15:09.377Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
6667d118d777e82ef9f093a3
Jiaxu Zou
admin_assigned
2024-06-11T10:15:18.617Z
643ac5720e5495afdefa1a51
/avatars/37fdbdca6e683f98cbbc6c048a9b86bc.svg
Jiaxu Zou
user
zoujx96
6667d118d777e82ef9f093a4
Michael Vasilkovsky
admin_assigned
2024-06-11T10:15:24.519Z
64775c34dbc2a416f8b648f6
/avatars/014b7011e0bd2e8ab6fa91a71ddb2d60.svg
Michael Vasilkovsky
user
waytobehigh
6667d118d777e82ef9f093a5
Vladislav Shakhrai
admin_assigned
2024-06-11T10:15:30.502Z
640290115caf6d21d67ea5f3
/avatars/236fcd4ab50e6305ca0f76317731a63c.svg
Vladislav Shakhrai
user
shakhrayv
6667d118d777e82ef9f093a6
Sergey Korolev
6667d118d777e82ef9f093a7
Sergey Tulyakov
6667d118d777e82ef9f093a8
Hsin-Ying Lee
admin_assigned
2024-06-11T10:15:53.329Z
631bbd918de8e645af71eed9
/avatars/1cb94deadc288d07c571c9289085548c.svg
Hsin-Ying Lee
user
james371507
GTR: Improving Large 3D Reconstruction Models through Geometry and
  Texture Refinement
We propose a novel approach for 3D mesh reconstruction from multi-view
images. Our method takes inspiration from large reconstruction models like LRM
that use a transformer-based triplane generator and a Neural Radiance Field
(NeRF) model trained on multi-view images. However, in our method, we introduce
several important modifications that allow us to significantly enhance 3D
reconstruction quality. First of all, we examine the original LRM architecture
and find several shortcomings. Subsequently, we introduce respective
modifications to the LRM architecture, which lead to improved multi-view image
representation and more computationally efficient training. Second, in order to
improve geometry reconstruction and enable supervision at full image
resolution, we extract meshes from the NeRF field in a differentiable manner
and fine-tune the NeRF model through mesh rendering. These modifications allow
us to achieve state-of-the-art performance on both 2D and 3D evaluation
metrics, such as a PSNR of 28.67 on Google Scanned Objects (GSO) dataset.
Despite these superior results, our feed-forward model still struggles to
reconstruct complex textures, such as text and portraits on assets. To address
this, we introduce a lightweight per-instance texture refinement procedure.
This procedure fine-tunes the triplane representation and the NeRF color
estimation model on the mesh surface using the input multi-view images in just
4 seconds. This refinement improves the PSNR to 29.79 and achieves faithful
reconstruction of complex textures, such as text. Additionally, our approach
enables various downstream applications, including text- or image-to-3D
generation.
The approach enhances 3D mesh reconstruction from multi-view images by modifying LRM architecture, improving NeRF model, and introducing a texture refinement procedure, achieving superior performance and enabling downstream applications.
3D mesh reconstruction multi-view images LRM transformer-based triplane generator Neural Radiance Field NeRF differentiable extraction mesh rendering PSNR Google Scanned Objects texture refinement text- or image-to-3D generation