title: Subject-Diffusion:Open Domain Personalized Text-to-Image Generation
    without Test-time Fine-tuning
summary: Recent progress in personalized image generation using diffusion models has
  been significant. However, development in the area of open-domain and
  non-fine-tuning personalized image generation is proceeding rather slowly. In
  this paper, we propose Subject-Diffusion, a novel open-domain personalized
  image generation model that, in addition to not requiring test-time
  fine-tuning, also only requires a single reference image to support
  personalized generation of single- or multi-subject in any domain. Firstly, we
  construct an automatic data labeling tool and use the LAION-Aesthetics dataset
  to construct a large-scale dataset consisting of 76M images and their
  corresponding subject detection bounding boxes, segmentation masks and text
  descriptions. Secondly, we design a new unified framework that combines text
  and image semantics by incorporating coarse location and fine-grained reference
  image control to maximize subject fidelity and generalization. Furthermore, we
  also adopt an attention control mechanism to support multi-subject generation.
  Extensive qualitative and quantitative results demonstrate that our method
  outperforms other SOTA frameworks in single, multiple, and human customized
  image generation. Please refer to our
  https://oppo-mente-lab.github.io/subject_diffusion/{project page}
paper_authors._id: 64bdd9674b4ff0d5095be4b5
paper_authors.name: Jian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-24T07:56:03.795Z
paper_authors._id: 63043115a362e7e8b51f47fa
paper_authors.avatarUrl: /avatars/d5c034262ddfcc3e7231b9a5abf461ad.svg
paper_authors.fullname: majian
paper_authors.type: user
paper_authors.user: majian0318
paper_authors._id: 64bdd9674b4ff0d5095be4b6
paper_authors.name: Junhao Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-24T07:55:16.747Z
paper_authors._id: 63426b3aaa45ed8ecf8fa9a8
paper_authors.avatarUrl: /avatars/529886aac934835b79e089580a1f0553.svg
paper_authors.fullname: Junhao Liang
paper_authors.type: user
paper_authors.user: Leeeung
paper_authors._id: 64bdd9674b4ff0d5095be4b7
paper_authors.name: Chen Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-24T14:15:21.318Z
paper_authors._id: 649d47e06503630c5269d93a
paper_authors.avatarUrl: /avatars/6812c9f79f2bd753ed68afb03baac202.svg
paper_authors.fullname: Chen Chen
paper_authors.type: user
paper_authors.user: beckhamchen
paper_authors._id: 64bdd9674b4ff0d5095be4b8
paper_authors.name: Haonan Lu
paper_title: Subject-Diffusion:Open Domain Personalized Text-to-Image Generation
    without Test-time Fine-tuning
paper_summary: Recent progress in personalized image generation using diffusion models has
  been significant. However, development in the area of open-domain and
  non-fine-tuning personalized image generation is proceeding rather slowly. In
  this paper, we propose Subject-Diffusion, a novel open-domain personalized
  image generation model that, in addition to not requiring test-time
  fine-tuning, also only requires a single reference image to support
  personalized generation of single- or multi-subject in any domain. Firstly, we
  construct an automatic data labeling tool and use the LAION-Aesthetics dataset
  to construct a large-scale dataset consisting of 76M images and their
  corresponding subject detection bounding boxes, segmentation masks and text
  descriptions. Secondly, we design a new unified framework that combines text
  and image semantics by incorporating coarse location and fine-grained reference
  image control to maximize subject fidelity and generalization. Furthermore, we
  also adopt an attention control mechanism to support multi-subject generation.
  Extensive qualitative and quantitative results demonstrate that our method
  outperforms other SOTA frameworks in single, multiple, and human customized
  image generation. Please refer to our
  https://oppo-mente-lab.github.io/subject_diffusion/{project page}
paper_ai_keywords: diffusion models Subject-Diffusion automatic data labeling tool LAION-Aesthetics dataset subject detection bounding boxes segmentation masks unified framework text semantics image semantics coarse location fine-grained reference image control subject fidelity generalization attention control mechanism multi-subject generation SOTA frameworks single image generation multiple image generation human customized image generation