Aligned Novel View Image and Geometry Synthesis via Cross-modal
  Attention Instillation
We introduce a diffusion-based framework that performs aligned novel view
image and geometry generation via a warping-and-inpainting methodology. Unlike
prior methods that require dense posed images or pose-embedded generative
models limited to in-domain views, our method leverages off-the-shelf geometry
predictors to predict partial geometries viewed from reference images, and
formulates novel-view synthesis as an inpainting task for both image and
geometry. To ensure accurate alignment between generated images and geometry,
we propose cross-modal attention distillation, where attention maps from the
image diffusion branch are injected into a parallel geometry diffusion branch
during both training and inference. This multi-task approach achieves
synergistic effects, facilitating geometrically robust image synthesis as well
as well-defined geometry prediction. We further introduce proximity-based mesh
conditioning to integrate depth and normal cues, interpolating between point
cloud and filtering erroneously predicted geometry from influencing the
generation process. Empirically, our method achieves high-fidelity
extrapolative view synthesis on both image and geometry across a range of
unseen scenes, delivers competitive reconstruction quality under interpolation
settings, and produces geometrically aligned colored point clouds for
comprehensive 3D completion. Project page is available at
https://cvlab-kaist.github.io/MoAI.
684faeba60b4a34dbe007ae2
Min-Seop Kwak
684faeba60b4a34dbe007ae3
Junho Kim
684faeba60b4a34dbe007ae4
Sangdoo Yun
admin_assigned
2025-06-17T07:59:45.315Z
648ac415718bc0670a5a5f56
/avatars/27189e289b808ef01689ff2abb7a56bf.svg
Sangdoo Yun
user
oodgnas
684faeba60b4a34dbe007ae5
Dongyoon Han
admin_assigned
2025-06-17T07:59:37.827Z
660f8cc1a61244f3df3d4426
/avatars/45d59766122bb3482f6dd7f9d98aa87a.svg
Dongyoon Han
user
calintz
684faeba60b4a34dbe007ae6
Taekyoung Kim
684faeba60b4a34dbe007ae7
Seungryong Kim
684faeba60b4a34dbe007ae8
Jin-Hwa Kim
admin_assigned
2025-06-17T07:59:24.415Z
651ed7b5755e92f7f12738b5
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0m82Fl5EAW4MQRjBQa94x.jpeg
Jin-Hwa Kim
user
jnhwkim
Aligned Novel View Image and Geometry Synthesis via Cross-modal
  Attention Instillation
We introduce a diffusion-based framework that performs aligned novel view
image and geometry generation via a warping-and-inpainting methodology. Unlike
prior methods that require dense posed images or pose-embedded generative
models limited to in-domain views, our method leverages off-the-shelf geometry
predictors to predict partial geometries viewed from reference images, and
formulates novel-view synthesis as an inpainting task for both image and
geometry. To ensure accurate alignment between generated images and geometry,
we propose cross-modal attention distillation, where attention maps from the
image diffusion branch are injected into a parallel geometry diffusion branch
during both training and inference. This multi-task approach achieves
synergistic effects, facilitating geometrically robust image synthesis as well
as well-defined geometry prediction. We further introduce proximity-based mesh
conditioning to integrate depth and normal cues, interpolating between point
cloud and filtering erroneously predicted geometry from influencing the
generation process. Empirically, our method achieves high-fidelity
extrapolative view synthesis on both image and geometry across a range of
unseen scenes, delivers competitive reconstruction quality under interpolation
settings, and produces geometrically aligned colored point clouds for
comprehensive 3D completion. Project page is available at
https://cvlab-kaist.github.io/MoAI.
A diffusion-based framework generates aligned novel views of images and geometry using warping-and-inpainting with cross-modal attention distillation and proximity-based mesh conditioning, achieving high-fidelity synthesis and 3D completion.
diffusion-based framework warping-and-inpainting off-the-shelf geometry predictors cross-modal attention distillation proximity-based mesh conditioning novel-view synthesis multi-task approach geometrically robust image synthesis well-defined geometry prediction extrapolative view synthesis 3D completion