title: DreamTuner: Single Image is Enough for Subject-Driven Generation
summary: Diffusion-based models have demonstrated impressive capabilities for
  text-to-image generation and are expected for personalized applications of
  subject-driven generation, which require the generation of customized concepts
  with one or a few reference images. However, existing methods based on
  fine-tuning fail to balance the trade-off between subject learning and the
  maintenance of the generation capabilities of pretrained models. Moreover,
  other methods that utilize additional image encoders tend to lose important
  details of the subject due to encoding compression. To address these
  challenges, we propose DreamTurner, a novel method that injects reference
  information from coarse to fine to achieve subject-driven image generation more
  effectively. DreamTurner introduces a subject-encoder for coarse subject
  identity preservation, where the compressed general subject features are
  introduced through an attention layer before visual-text cross-attention. We
  then modify the self-attention layers within pretrained text-to-image models to
  self-subject-attention layers to refine the details of the target subject. The
  generated image queries detailed features from both the reference image and
  itself in self-subject-attention. It is worth emphasizing that
  self-subject-attention is an effective, elegant, and training-free method for
  maintaining the detailed features of customized subjects and can serve as a
  plug-and-play solution during inference. Finally, with additional
  subject-driven fine-tuning, DreamTurner achieves remarkable performance in
  subject-driven image generation, which can be controlled by a text or other
  conditions such as pose. For further details, please visit the project page at
  https://dreamtuner-diffusion.github.io/.
paper_authors._id: 6584ef1fbae07363659afa17
paper_authors.name: Miao Hua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T11:57:16.145Z
paper_authors._id: 6309807c03837fbcf0856059
paper_authors.avatarUrl: /avatars/fcafe6726e33094fed463a48f04f044f.svg
paper_authors.fullname: miaohua
paper_authors.type: user
paper_authors.user: miaohua
paper_authors._id: 6584ef1fbae07363659afa18
paper_authors.name: Jiawei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T14:24:26.589Z
paper_authors._id: 63049b95dae2eb7d083f1bf3
paper_authors.avatarUrl: /avatars/5eaeadc1318724b54ea59873da11275f.svg
paper_authors.fullname: Jiawei Liu
paper_authors.type: user
paper_authors.user: jwliu-cc
paper_authors._id: 6584ef1fbae07363659afa19
paper_authors.name: Fei Ding
paper_authors._id: 6584ef1fbae07363659afa1a
paper_authors.name: Wei Liu
paper_authors._id: 6584ef1fbae07363659afa1b
paper_authors.name: Jie Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-22T07:35:54.057Z
paper_authors._id: 6381c5d63680a7cf34e08ca9
paper_authors.avatarUrl: /avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
paper_authors.fullname: wujie10558@gmail.com
paper_authors.type: user
paper_authors.user: wujie10
paper_authors._id: 6584ef1fbae07363659afa1c
paper_authors.name: Qian He
paper_title: DreamTuner: Single Image is Enough for Subject-Driven Generation
paper_summary: Diffusion-based models have demonstrated impressive capabilities for
  text-to-image generation and are expected for personalized applications of
  subject-driven generation, which require the generation of customized concepts
  with one or a few reference images. However, existing methods based on
  fine-tuning fail to balance the trade-off between subject learning and the
  maintenance of the generation capabilities of pretrained models. Moreover,
  other methods that utilize additional image encoders tend to lose important
  details of the subject due to encoding compression. To address these
  challenges, we propose DreamTurner, a novel method that injects reference
  information from coarse to fine to achieve subject-driven image generation more
  effectively. DreamTurner introduces a subject-encoder for coarse subject
  identity preservation, where the compressed general subject features are
  introduced through an attention layer before visual-text cross-attention. We
  then modify the self-attention layers within pretrained text-to-image models to
  self-subject-attention layers to refine the details of the target subject. The
  generated image queries detailed features from both the reference image and
  itself in self-subject-attention. It is worth emphasizing that
  self-subject-attention is an effective, elegant, and training-free method for
  maintaining the detailed features of customized subjects and can serve as a
  plug-and-play solution during inference. Finally, with additional
  subject-driven fine-tuning, DreamTurner achieves remarkable performance in
  subject-driven image generation, which can be controlled by a text or other
  conditions such as pose. For further details, please visit the project page at
  https://dreamtuner-diffusion.github.io/.
paper_ai_summary: DreamTurner uses a novel approach by injecting reference information through subject encoders and self-subject-attention layers to enhance subject-driven image generation, balancing subject learning and model capabilities.
paper_ai_keywords: diffusion-based models text-to-image generation fine-tuning subject-driven generation subject-encoder attention layer visual-text cross-attention self-attention layers self-subject-attention subject-driven fine-tuning