title: DreamComposer: Controllable 3D Object Generation via Multi-View
    Conditions
summary: Utilizing pre-trained 2D large-scale generative models, recent works are
  capable of generating high-quality novel views from a single in-the-wild image.
  However, due to the lack of information from multiple views, these works
  encounter difficulties in generating controllable novel views. In this paper,
  we present DreamComposer, a flexible and scalable framework that can enhance
  existing view-aware diffusion models by injecting multi-view conditions.
  Specifically, DreamComposer first uses a view-aware 3D lifting module to obtain
  3D representations of an object from multiple views. Then, it renders the
  latent features of the target view from 3D representations with the multi-view
  feature fusion module. Finally the target view features extracted from
  multi-view inputs are injected into a pre-trained diffusion model. Experiments
  show that DreamComposer is compatible with state-of-the-art diffusion models
  for zero-shot novel view synthesis, further enhancing them to generate
  high-fidelity novel view images with multi-view conditions, ready for
  controllable 3D object reconstruction and various other applications.
paper_authors._id: 657140ccb6f7eb7f31177890
paper_authors.name: Yunhan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:12:25.009Z
paper_authors._id: 64b4eecf2fc8324fcb63b404
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b4eecf2fc8324fcb63b404/zGYqYVB4-o-GBMybJ8CDA.png
paper_authors.fullname: Yunhan Yang
paper_authors.type: user
paper_authors.user: yhyang-myron
paper_authors._id: 657140ccb6f7eb7f31177891
paper_authors.name: Yukun Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T08:26:39.818Z
paper_authors._id: 638ee900ee7e45e0474a5712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638ee900ee7e45e0474a5712/KLli_eCbWwffKR7oLDmV3.jpeg
paper_authors.fullname: Yukun Huang
paper_authors.type: user
paper_authors.user: KevinHuang
paper_authors._id: 657140ccb6f7eb7f31177892
paper_authors.name: Xiaoyang Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:14:12.843Z
paper_authors._id: 643e5d6a1d0e956d94bb3608
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643e5d6a1d0e956d94bb3608/mTnu7Dts2RmDklHlp9Gqu.jpeg
paper_authors.fullname: Xiaoyang Wu
paper_authors.type: user
paper_authors.user: Gofinge
paper_authors._id: 657140ccb6f7eb7f31177893
paper_authors.name: Yuan-Chen Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:14:19.747Z
paper_authors._id: 6346aaa3f06b237ba4e297b0
paper_authors.avatarUrl: /avatars/5acb986e993eab1461200f3e9d99d022.svg
paper_authors.fullname: Yuan-Chen Guo
paper_authors.type: user
paper_authors.user: bennyguo
paper_authors._id: 657140ccb6f7eb7f31177894
paper_authors.name: Song-Hai Zhang
paper_authors._id: 657140ccb6f7eb7f31177895
paper_authors.name: Hengshuang Zhao
paper_authors._id: 657140ccb6f7eb7f31177896
paper_authors.name: Tong He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T12:15:14.826Z
paper_authors._id: 63184606ab1bf092d6f059be
paper_authors.avatarUrl: /avatars/b49d125d112a01106b48a2fcda481c3e.svg
paper_authors.fullname: Tong He
paper_authors.type: user
paper_authors.user: hetong007
paper_authors._id: 657140ccb6f7eb7f31177897
paper_authors.name: Xihui Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-03T18:30:14.921Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_title: DreamComposer: Controllable 3D Object Generation via Multi-View
    Conditions
paper_summary: Utilizing pre-trained 2D large-scale generative models, recent works are
  capable of generating high-quality novel views from a single in-the-wild image.
  However, due to the lack of information from multiple views, these works
  encounter difficulties in generating controllable novel views. In this paper,
  we present DreamComposer, a flexible and scalable framework that can enhance
  existing view-aware diffusion models by injecting multi-view conditions.
  Specifically, DreamComposer first uses a view-aware 3D lifting module to obtain
  3D representations of an object from multiple views. Then, it renders the
  latent features of the target view from 3D representations with the multi-view
  feature fusion module. Finally the target view features extracted from
  multi-view inputs are injected into a pre-trained diffusion model. Experiments
  show that DreamComposer is compatible with state-of-the-art diffusion models
  for zero-shot novel view synthesis, further enhancing them to generate
  high-fidelity novel view images with multi-view conditions, ready for
  controllable 3D object reconstruction and various other applications.
paper_ai_summary: DreamComposer enhances diffusion models for novel view synthesis by integrating multi-view conditions to improve image quality and controllability.
paper_ai_keywords: view-aware diffusion models 3D lifting module multi-view feature fusion module novel view synthesis high-fidelity novel view images 3D object reconstruction