MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes
Repurposing pre-trained diffusion models has been proven to be effective for
NVS. However, these methods are mostly limited to a single object; directly
applying such methods to compositional multi-object scenarios yields inferior
results, especially incorrect object placement and inconsistent shape and
appearance under novel views. How to enhance and systematically evaluate the
cross-view consistency of such models remains under-explored. To address this
issue, we propose MOVIS to enhance the structural awareness of the
view-conditioned diffusion model for multi-object NVS in terms of model inputs,
auxiliary tasks, and training strategy. First, we inject structure-aware
features, including depth and object mask, into the denoising U-Net to enhance
the model's comprehension of object instances and their spatial relationships.
Second, we introduce an auxiliary task requiring the model to simultaneously
predict novel view object masks, further improving the model's capability in
differentiating and placing objects. Finally, we conduct an in-depth analysis
of the diffusion sampling process and carefully devise a structure-guided
timestep sampling scheduler during training, which balances the learning of
global object placement and fine-grained detail recovery. To systematically
evaluate the plausibility of synthesized images, we propose to assess
cross-view consistency and novel view object placement alongside existing
image-level NVS metrics. Extensive experiments on challenging synthetic and
realistic datasets demonstrate that our method exhibits strong generalization
capabilities and produces consistent novel view synthesis, highlighting its
potential to guide future 3D-aware multi-object NVS tasks.
6760ef0da38bf4e171b6eb31
Ruijie Lu
admin_assigned
2024-12-17T11:14:33.511Z
64ab8cb76324705e6a65f7c4
/avatars/15dcae6c345d31ea6e17c11108a7deb7.svg
Ruijie Lu
user
JasonAplp
6760ef0da38bf4e171b6eb32
Yixin Chen
admin_assigned
2024-12-17T11:14:27.475Z
64c396def6fe448b1ad553d6
/avatars/b2ce4739f42dc00ee974fff7ee1cb301.svg
Yixin Chen
user
YixinChen
6760ef0da38bf4e171b6eb33
Junfeng Ni
admin_assigned
2024-12-17T11:14:22.016Z
65ae5edddacd99fd58277620
https://cdn-avatars.huggingface.co/v1/production/uploads/65ae5edddacd99fd58277620/Yq2adEKtkxaSzdEhCfpnY.jpeg
Junfeng Ni
user
JunfengNi
6760ef0da38bf4e171b6eb34
Baoxiong Jia
admin_assigned
2024-12-17T11:14:15.760Z
6304b389bad6ce7fc02691d5
/avatars/a762ca59624ce409650165f36b973488.svg
Baoxiong Jia
user
BuzzBeater
6760ef0da38bf4e171b6eb35
Yu Liu
admin_assigned
2024-12-17T11:14:09.545Z
636de85cc4a7a729c164d2b5
/avatars/3e281e547e1697e1c06805e7e63f3918.svg
Yu Liu
user
YuLiu
6760ef0da38bf4e171b6eb36
Diwen Wan
admin_assigned
2024-12-17T11:13:55.269Z
63fef01cc0ec83fda43d5c40
/avatars/363efd5c58e4e810267b666240e3f6b3.svg
Diwen Wan
user
dnvtmf
6760ef0da38bf4e171b6eb37
Gang Zeng
6760ef0da38bf4e171b6eb38
Siyuan Huang
admin_assigned
2024-12-17T11:13:36.914Z
63c7a33121bd95f80ed74652
/avatars/7dd59afea785a2bff0ec2b757abd474e.svg
Siyuan Huang
user
thuhsy
MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes
Repurposing pre-trained diffusion models has been proven to be effective for
NVS. However, these methods are mostly limited to a single object; directly
applying such methods to compositional multi-object scenarios yields inferior
results, especially incorrect object placement and inconsistent shape and
appearance under novel views. How to enhance and systematically evaluate the
cross-view consistency of such models remains under-explored. To address this
issue, we propose MOVIS to enhance the structural awareness of the
view-conditioned diffusion model for multi-object NVS in terms of model inputs,
auxiliary tasks, and training strategy. First, we inject structure-aware
features, including depth and object mask, into the denoising U-Net to enhance
the model's comprehension of object instances and their spatial relationships.
Second, we introduce an auxiliary task requiring the model to simultaneously
predict novel view object masks, further improving the model's capability in
differentiating and placing objects. Finally, we conduct an in-depth analysis
of the diffusion sampling process and carefully devise a structure-guided
timestep sampling scheduler during training, which balances the learning of
global object placement and fine-grained detail recovery. To systematically
evaluate the plausibility of synthesized images, we propose to assess
cross-view consistency and novel view object placement alongside existing
image-level NVS metrics. Extensive experiments on challenging synthetic and
realistic datasets demonstrate that our method exhibits strong generalization
capabilities and produces consistent novel view synthesis, highlighting its
potential to guide future 3D-aware multi-object NVS tasks.
MOVIS enhances structure-awareness in view-conditioned diffusion models for multi-object NVS by incorporating depth and object mask features, auxiliary mask prediction, and structure-guided timestep sampling, leading to better cross-view consistency and object placement.
diffusion models NVS multi-object NVS view-conditioned diffusion model denoising U-Net depth object mask auxiliary task structure-guided timestep sampling scheduler cross-view consistency novel view object placement image-level NVS metrics generalization capabilities