MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation
This paper introduces MIDI, a novel paradigm for compositional 3D scene
generation from a single image. Unlike existing methods that rely on
reconstruction or retrieval techniques or recent approaches that employ
multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D
object generation models to multi-instance diffusion models, enabling the
simultaneous generation of multiple 3D instances with accurate spatial
relationships and high generalizability. At its core, MIDI incorporates a novel
multi-instance attention mechanism, that effectively captures inter-object
interactions and spatial coherence directly within the generation process,
without the need for complex multi-step processes. The method utilizes partial
object images and global scene context as inputs, directly modeling object
completion during 3D generation. During training, we effectively supervise the
interactions between 3D instances using a limited amount of scene-level data,
while incorporating single-object data for regularization, thereby maintaining
the pre-trained generalization ability. MIDI demonstrates state-of-the-art
performance in image-to-scene generation, validated through evaluations on
synthetic data, real-world scene data, and stylized scene images generated by
text-to-image diffusion models.
67511976753718a53ff7fdfb
Zehuan Huang
claimed_verified
2024-12-05T09:18:15.519Z
6375d136dee28348a9c63cbf
https://cdn-avatars.huggingface.co/v1/production/uploads/6375d136dee28348a9c63cbf/gK465HBrQWIOZ-qtHb-Vh.jpeg
zehuan-huang
user
huanngzh
67511976753718a53ff7fdfc
Yuan-Chen Guo
admin_assigned
2024-12-05T10:45:34.099Z
6346aaa3f06b237ba4e297b0
/avatars/5acb986e993eab1461200f3e9d99d022.svg
Yuan-Chen Guo
user
bennyguo
67511976753718a53ff7fdfd
Xingqiao An
admin_assigned
2024-12-05T10:45:39.396Z
674d7d6fe3908280620d1ac4
/avatars/29157f5a208937655c3f5e5f2ac23c6a.svg
Xingqiao_An
user
XingqiaoAn
67511976753718a53ff7fdfe
Yunhan Yang
admin_assigned
2024-12-05T10:45:45.545Z
64b4eecf2fc8324fcb63b404
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4eecf2fc8324fcb63b404/zGYqYVB4-o-GBMybJ8CDA.png
Yunhan Yang
user
yhyang-myron
67511976753718a53ff7fdff
Yangguang Li
claimed_verified
2025-02-14T16:20:16.593Z
64d71083a787c9bc7b9f1238
/avatars/d0b0546dec7fc5792921154bec41385a.svg
Yangguang Li
user
Lp256
67511976753718a53ff7fe00
Zi-Xin Zou
admin_assigned
2024-12-05T10:46:03.490Z
644dbf6453ad80c6593bf748
/avatars/0e170cf2aa8d7f0f3f83e36f06f023f8.svg
Zixin Zou
user
zouzx
67511976753718a53ff7fe01
Ding Liang
67511976753718a53ff7fe02
Xihui Liu
admin_assigned
2024-12-05T10:46:12.847Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
67511976753718a53ff7fe03
Yan-Pei Cao
admin_assigned
2024-12-05T10:46:20.698Z
638066faf022c8a5803f7eb8
/avatars/4cfd699c3f6c5461b12b7dc5e3fe183d.svg
Yanpei Cao
user
pookiefoof
67511976753718a53ff7fe04
Lu Sheng
claimed_verified
2024-12-06T15:57:38.089Z
65b722dbe02a17f0f8d1cc6b
/avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
Lu Sheng
user
lsheng2024
MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation
This paper introduces MIDI, a novel paradigm for compositional 3D scene
generation from a single image. Unlike existing methods that rely on
reconstruction or retrieval techniques or recent approaches that employ
multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D
object generation models to multi-instance diffusion models, enabling the
simultaneous generation of multiple 3D instances with accurate spatial
relationships and high generalizability. At its core, MIDI incorporates a novel
multi-instance attention mechanism, that effectively captures inter-object
interactions and spatial coherence directly within the generation process,
without the need for complex multi-step processes. The method utilizes partial
object images and global scene context as inputs, directly modeling object
completion during 3D generation. During training, we effectively supervise the
interactions between 3D instances using a limited amount of scene-level data,
while incorporating single-object data for regularization, thereby maintaining
the pre-trained generalization ability. MIDI demonstrates state-of-the-art
performance in image-to-scene generation, validated through evaluations on
synthetic data, real-world scene data, and stylized scene images generated by
text-to-image diffusion models.
MIDI is a novel method for 3D scene generation from a single image using multi-instance diffusion models with attention mechanisms, achieving state-of-the-art results across different datasets.
image-to-3D object generation models diffusion models multi-instance attention mechanism partial object images global scene context scene-level data single-object data