title: MV-Adapter: Multi-view Consistent Image Generation Made Easy
summary: Existing multi-view image generation methods often make invasive
  modifications to pre-trained text-to-image (T2I) models and require full
  fine-tuning, leading to (1) high computational costs, especially with large
  base models and high-resolution images, and (2) degradation in image quality
  due to optimization difficulties and scarce high-quality 3D data. In this
  paper, we propose the first adapter-based solution for multi-view image
  generation, and introduce MV-Adapter, a versatile plug-and-play adapter that
  enhances T2I models and their derivatives without altering the original network
  structure or feature space. By updating fewer parameters, MV-Adapter enables
  efficient training and preserves the prior knowledge embedded in pre-trained
  models, mitigating overfitting risks. To efficiently model the 3D geometric
  knowledge within the adapter, we introduce innovative designs that include
  duplicated self-attention layers and parallel attention architecture, enabling
  the adapter to inherit the powerful priors of the pre-trained models to model
  the novel 3D knowledge. Moreover, we present a unified condition encoder that
  seamlessly integrates camera parameters and geometric information, facilitating
  applications such as text- and image-based 3D generation and texturing.
  MV-Adapter achieves multi-view generation at 768 resolution on Stable Diffusion
  XL (SDXL), and demonstrates adaptability and versatility. It can also be
  extended to arbitrary view generation, enabling broader applications. We
  demonstrate that MV-Adapter sets a new quality standard for multi-view image
  generation, and opens up new possibilities due to its efficiency, adaptability
  and versatility.
paper_authors._id: 6752727b61e5a1edbc6f45fc
paper_authors.name: Zehuan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T10:41:33.167Z
paper_authors._id: 6375d136dee28348a9c63cbf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6375d136dee28348a9c63cbf/gK465HBrQWIOZ-qtHb-Vh.jpeg
paper_authors.fullname: zehuan-huang
paper_authors.type: user
paper_authors.user: huanngzh
paper_authors._id: 6752727b61e5a1edbc6f45fd
paper_authors.name: Yuan-Chen Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T10:41:38.686Z
paper_authors._id: 6346aaa3f06b237ba4e297b0
paper_authors.avatarUrl: /avatars/5acb986e993eab1461200f3e9d99d022.svg
paper_authors.fullname: Yuan-Chen Guo
paper_authors.type: user
paper_authors.user: bennyguo
paper_authors._id: 6752727b61e5a1edbc6f45fe
paper_authors.name: Haoran Wang
paper_authors._id: 6752727b61e5a1edbc6f45ff
paper_authors.name: Ran Yi
paper_authors._id: 6752727b61e5a1edbc6f4600
paper_authors.name: Lizhuang Ma
paper_authors._id: 6752727b61e5a1edbc6f4601
paper_authors.name: Yan-Pei Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T10:42:23.272Z
paper_authors._id: 638066faf022c8a5803f7eb8
paper_authors.avatarUrl: /avatars/4cfd699c3f6c5461b12b7dc5e3fe183d.svg
paper_authors.fullname: Yanpei Cao
paper_authors.type: user
paper_authors.user: pookiefoof
paper_authors._id: 6752727b61e5a1edbc6f4602
paper_authors.name: Lu Sheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:15.578Z
paper_authors._id: 65b722dbe02a17f0f8d1cc6b
paper_authors.avatarUrl: /avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
paper_authors.fullname: Lu Sheng
paper_authors.type: user
paper_authors.user: lsheng2024
paper_title: MV-Adapter: Multi-view Consistent Image Generation Made Easy
paper_summary: Existing multi-view image generation methods often make invasive
  modifications to pre-trained text-to-image (T2I) models and require full
  fine-tuning, leading to (1) high computational costs, especially with large
  base models and high-resolution images, and (2) degradation in image quality
  due to optimization difficulties and scarce high-quality 3D data. In this
  paper, we propose the first adapter-based solution for multi-view image
  generation, and introduce MV-Adapter, a versatile plug-and-play adapter that
  enhances T2I models and their derivatives without altering the original network
  structure or feature space. By updating fewer parameters, MV-Adapter enables
  efficient training and preserves the prior knowledge embedded in pre-trained
  models, mitigating overfitting risks. To efficiently model the 3D geometric
  knowledge within the adapter, we introduce innovative designs that include
  duplicated self-attention layers and parallel attention architecture, enabling
  the adapter to inherit the powerful priors of the pre-trained models to model
  the novel 3D knowledge. Moreover, we present a unified condition encoder that
  seamlessly integrates camera parameters and geometric information, facilitating
  applications such as text- and image-based 3D generation and texturing.
  MV-Adapter achieves multi-view generation at 768 resolution on Stable Diffusion
  XL (SDXL), and demonstrates adaptability and versatility. It can also be
  extended to arbitrary view generation, enabling broader applications. We
  demonstrate that MV-Adapter sets a new quality standard for multi-view image
  generation, and opens up new possibilities due to its efficiency, adaptability
  and versatility.
paper_ai_keywords: adapter-based solution duplicated self-attention layers parallel attention architecture condition encoder text-to-image (T2I) models multi-view image generation parameter-efficient fine-tuning pre-trained models overfitting risks 3D geometric knowledge camera parameters geometric information text- and image-based 3D generation texturing arbitrary view generation Stable Diffusion XL (SDXL)