title: MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes
summary: Repurposing pre-trained diffusion models has been proven to be effective for
  NVS. However, these methods are mostly limited to a single object; directly
  applying such methods to compositional multi-object scenarios yields inferior
  results, especially incorrect object placement and inconsistent shape and
  appearance under novel views. How to enhance and systematically evaluate the
  cross-view consistency of such models remains under-explored. To address this
  issue, we propose MOVIS to enhance the structural awareness of the
  view-conditioned diffusion model for multi-object NVS in terms of model inputs,
  auxiliary tasks, and training strategy. First, we inject structure-aware
  features, including depth and object mask, into the denoising U-Net to enhance
  the model's comprehension of object instances and their spatial relationships.
  Second, we introduce an auxiliary task requiring the model to simultaneously
  predict novel view object masks, further improving the model's capability in
  differentiating and placing objects. Finally, we conduct an in-depth analysis
  of the diffusion sampling process and carefully devise a structure-guided
  timestep sampling scheduler during training, which balances the learning of
  global object placement and fine-grained detail recovery. To systematically
  evaluate the plausibility of synthesized images, we propose to assess
  cross-view consistency and novel view object placement alongside existing
  image-level NVS metrics. Extensive experiments on challenging synthetic and
  realistic datasets demonstrate that our method exhibits strong generalization
  capabilities and produces consistent novel view synthesis, highlighting its
  potential to guide future 3D-aware multi-object NVS tasks.
paper_authors._id: 6760ef0da38bf4e171b6eb31
paper_authors.name: Ruijie Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:14:33.511Z
paper_authors._id: 64ab8cb76324705e6a65f7c4
paper_authors.avatarUrl: /avatars/15dcae6c345d31ea6e17c11108a7deb7.svg
paper_authors.fullname: Ruijie Lu
paper_authors.type: user
paper_authors.user: JasonAplp
paper_authors._id: 6760ef0da38bf4e171b6eb32
paper_authors.name: Yixin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:14:27.475Z
paper_authors._id: 64c396def6fe448b1ad553d6
paper_authors.avatarUrl: /avatars/b2ce4739f42dc00ee974fff7ee1cb301.svg
paper_authors.fullname: Yixin Chen
paper_authors.type: user
paper_authors.user: YixinChen
paper_authors._id: 6760ef0da38bf4e171b6eb33
paper_authors.name: Junfeng Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:14:22.016Z
paper_authors._id: 65ae5edddacd99fd58277620
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65ae5edddacd99fd58277620/Yq2adEKtkxaSzdEhCfpnY.jpeg
paper_authors.fullname: Junfeng Ni
paper_authors.type: user
paper_authors.user: JunfengNi
paper_authors._id: 6760ef0da38bf4e171b6eb34
paper_authors.name: Baoxiong Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:14:15.760Z
paper_authors._id: 6304b389bad6ce7fc02691d5
paper_authors.avatarUrl: /avatars/a762ca59624ce409650165f36b973488.svg
paper_authors.fullname: Baoxiong Jia
paper_authors.type: user
paper_authors.user: BuzzBeater
paper_authors._id: 6760ef0da38bf4e171b6eb35
paper_authors.name: Yu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:14:09.545Z
paper_authors._id: 636de85cc4a7a729c164d2b5
paper_authors.avatarUrl: /avatars/3e281e547e1697e1c06805e7e63f3918.svg
paper_authors.fullname: Yu Liu
paper_authors.type: user
paper_authors.user: YuLiu
paper_authors._id: 6760ef0da38bf4e171b6eb36
paper_authors.name: Diwen Wan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:13:55.269Z
paper_authors._id: 63fef01cc0ec83fda43d5c40
paper_authors.avatarUrl: /avatars/363efd5c58e4e810267b666240e3f6b3.svg
paper_authors.fullname: Diwen Wan
paper_authors.type: user
paper_authors.user: dnvtmf
paper_authors._id: 6760ef0da38bf4e171b6eb37
paper_authors.name: Gang Zeng
paper_authors._id: 6760ef0da38bf4e171b6eb38
paper_authors.name: Siyuan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:13:36.914Z
paper_authors._id: 63c7a33121bd95f80ed74652
paper_authors.avatarUrl: /avatars/7dd59afea785a2bff0ec2b757abd474e.svg
paper_authors.fullname: Siyuan Huang
paper_authors.type: user
paper_authors.user: thuhsy
paper_title: MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes
paper_summary: Repurposing pre-trained diffusion models has been proven to be effective for
  NVS. However, these methods are mostly limited to a single object; directly
  applying such methods to compositional multi-object scenarios yields inferior
  results, especially incorrect object placement and inconsistent shape and
  appearance under novel views. How to enhance and systematically evaluate the
  cross-view consistency of such models remains under-explored. To address this
  issue, we propose MOVIS to enhance the structural awareness of the
  view-conditioned diffusion model for multi-object NVS in terms of model inputs,
  auxiliary tasks, and training strategy. First, we inject structure-aware
  features, including depth and object mask, into the denoising U-Net to enhance
  the model's comprehension of object instances and their spatial relationships.
  Second, we introduce an auxiliary task requiring the model to simultaneously
  predict novel view object masks, further improving the model's capability in
  differentiating and placing objects. Finally, we conduct an in-depth analysis
  of the diffusion sampling process and carefully devise a structure-guided
  timestep sampling scheduler during training, which balances the learning of
  global object placement and fine-grained detail recovery. To systematically
  evaluate the plausibility of synthesized images, we propose to assess
  cross-view consistency and novel view object placement alongside existing
  image-level NVS metrics. Extensive experiments on challenging synthetic and
  realistic datasets demonstrate that our method exhibits strong generalization
  capabilities and produces consistent novel view synthesis, highlighting its
  potential to guide future 3D-aware multi-object NVS tasks.
paper_ai_summary: MOVIS enhances structure-awareness in view-conditioned diffusion models for multi-object NVS by incorporating depth and object mask features, auxiliary mask prediction, and structure-guided timestep sampling, leading to better cross-view consistency and object placement.
paper_ai_keywords: diffusion models NVS multi-object NVS view-conditioned diffusion model denoising U-Net depth object mask auxiliary task structure-guided timestep sampling scheduler cross-view consistency novel view object placement image-level NVS metrics generalization capabilities