GMT: Enhancing Generalizable Neural Rendering via Geometry-Driven Multi-Reference Texture Transfer
<|reference_start|>GMT: Enhancing Generalizable Neural Rendering via Geometry-Driven Multi-Reference Texture Transfer: Novel view synthesis (NVS) aims to generate images at arbitrary viewpoints using multi-view images, and recent insights from neural radiance fields (NeRF) have contributed to remarkable improvements. Recently, studies on generalizable NeRF (G-NeRF) have addressed the challenge of per-scene optimization in NeRFs. The construction of radiance fields on-the-fly in G-NeRF simplifies the NVS process, making it well-suited for real-world applications. Meanwhile, G-NeRF still struggles in representing fine details for a specific scene due to the absence of per-scene optimization, even with texture-rich multi-view source inputs. As a remedy, we propose a Geometry-driven Multi-reference Texture transfer network (GMT) available as a plug-and-play module designed for G-NeRF. Specifically, we propose ray-imposed deformable convolution (RayDCN), which aligns input and reference features reflecting scene geometry. Additionally, the proposed texture preserving transformer (TP-Former) aggregates multi-view source features while preserving texture information. Consequently, our module enables direct interaction between adjacent pixels during the image enhancement process, which is deficient in G-NeRF models with an independent rendering process per pixel. This addresses constraints that hinder the ability to capture high-frequency details. Experiments show that our plug-and-play module consistently improves G-NeRF models on various benchmark datasets.<|reference_end|>
@article{yoon2024gmt:,
    title={GMT: Enhancing Generalizable Neural Rendering via Geometry-Driven
  Multi-Reference Texture Transfer},
    author={Youngho Yoon, Hyun-Kurl Jang, and Kuk-Jin Yoon},
    journal={arXiv preprint arXiv:2410.00672},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.00672},
    primaryClass={cs.CV}
}