title: MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation
summary: This paper introduces MIDI, a novel paradigm for compositional 3D scene
  generation from a single image. Unlike existing methods that rely on
  reconstruction or retrieval techniques or recent approaches that employ
  multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D
  object generation models to multi-instance diffusion models, enabling the
  simultaneous generation of multiple 3D instances with accurate spatial
  relationships and high generalizability. At its core, MIDI incorporates a novel
  multi-instance attention mechanism, that effectively captures inter-object
  interactions and spatial coherence directly within the generation process,
  without the need for complex multi-step processes. The method utilizes partial
  object images and global scene context as inputs, directly modeling object
  completion during 3D generation. During training, we effectively supervise the
  interactions between 3D instances using a limited amount of scene-level data,
  while incorporating single-object data for regularization, thereby maintaining
  the pre-trained generalization ability. MIDI demonstrates state-of-the-art
  performance in image-to-scene generation, validated through evaluations on
  synthetic data, real-world scene data, and stylized scene images generated by
  text-to-image diffusion models.
paper_authors._id: 67511976753718a53ff7fdfb
paper_authors.name: Zehuan Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:18:15.519Z
paper_authors._id: 6375d136dee28348a9c63cbf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6375d136dee28348a9c63cbf/gK465HBrQWIOZ-qtHb-Vh.jpeg
paper_authors.fullname: zehuan-huang
paper_authors.type: user
paper_authors.user: huanngzh
paper_authors._id: 67511976753718a53ff7fdfc
paper_authors.name: Yuan-Chen Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:45:34.099Z
paper_authors._id: 6346aaa3f06b237ba4e297b0
paper_authors.avatarUrl: /avatars/5acb986e993eab1461200f3e9d99d022.svg
paper_authors.fullname: Yuan-Chen Guo
paper_authors.type: user
paper_authors.user: bennyguo
paper_authors._id: 67511976753718a53ff7fdfd
paper_authors.name: Xingqiao An
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:45:39.396Z
paper_authors._id: 674d7d6fe3908280620d1ac4
paper_authors.avatarUrl: /avatars/29157f5a208937655c3f5e5f2ac23c6a.svg
paper_authors.fullname: Xingqiao_An
paper_authors.type: user
paper_authors.user: XingqiaoAn
paper_authors._id: 67511976753718a53ff7fdfe
paper_authors.name: Yunhan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:45:45.545Z
paper_authors._id: 64b4eecf2fc8324fcb63b404
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b4eecf2fc8324fcb63b404/zGYqYVB4-o-GBMybJ8CDA.png
paper_authors.fullname: Yunhan Yang
paper_authors.type: user
paper_authors.user: yhyang-myron
paper_authors._id: 67511976753718a53ff7fdff
paper_authors.name: Yangguang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T16:20:16.593Z
paper_authors._id: 64d71083a787c9bc7b9f1238
paper_authors.avatarUrl: /avatars/d0b0546dec7fc5792921154bec41385a.svg
paper_authors.fullname: Yangguang Li
paper_authors.type: user
paper_authors.user: Lp256
paper_authors._id: 67511976753718a53ff7fe00
paper_authors.name: Zi-Xin Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:46:03.490Z
paper_authors._id: 644dbf6453ad80c6593bf748
paper_authors.avatarUrl: /avatars/0e170cf2aa8d7f0f3f83e36f06f023f8.svg
paper_authors.fullname: Zixin Zou
paper_authors.type: user
paper_authors.user: zouzx
paper_authors._id: 67511976753718a53ff7fe01
paper_authors.name: Ding Liang
paper_authors._id: 67511976753718a53ff7fe02
paper_authors.name: Xihui Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:46:12.847Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_authors._id: 67511976753718a53ff7fe03
paper_authors.name: Yan-Pei Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:46:20.698Z
paper_authors._id: 638066faf022c8a5803f7eb8
paper_authors.avatarUrl: /avatars/4cfd699c3f6c5461b12b7dc5e3fe183d.svg
paper_authors.fullname: Yanpei Cao
paper_authors.type: user
paper_authors.user: pookiefoof
paper_authors._id: 67511976753718a53ff7fe04
paper_authors.name: Lu Sheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:38.089Z
paper_authors._id: 65b722dbe02a17f0f8d1cc6b
paper_authors.avatarUrl: /avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
paper_authors.fullname: Lu Sheng
paper_authors.type: user
paper_authors.user: lsheng2024
paper_title: MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation
paper_summary: This paper introduces MIDI, a novel paradigm for compositional 3D scene
  generation from a single image. Unlike existing methods that rely on
  reconstruction or retrieval techniques or recent approaches that employ
  multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D
  object generation models to multi-instance diffusion models, enabling the
  simultaneous generation of multiple 3D instances with accurate spatial
  relationships and high generalizability. At its core, MIDI incorporates a novel
  multi-instance attention mechanism, that effectively captures inter-object
  interactions and spatial coherence directly within the generation process,
  without the need for complex multi-step processes. The method utilizes partial
  object images and global scene context as inputs, directly modeling object
  completion during 3D generation. During training, we effectively supervise the
  interactions between 3D instances using a limited amount of scene-level data,
  while incorporating single-object data for regularization, thereby maintaining
  the pre-trained generalization ability. MIDI demonstrates state-of-the-art
  performance in image-to-scene generation, validated through evaluations on
  synthetic data, real-world scene data, and stylized scene images generated by
  text-to-image diffusion models.
paper_ai_keywords: multi-instance diffusion models attention mechanism inter-object interactions spatial coherence image-to-3D object generation single-object data for regularization scene-level data text-to-image diffusion models