title: BootPIG: Bootstrapping Zero-shot Personalized Image Generation
    Capabilities in Pretrained Diffusion Models
summary: Recent text-to-image generation models have demonstrated incredible success
  in generating images that faithfully follow input prompts. However, the
  requirement of using words to describe a desired concept provides limited
  control over the appearance of the generated concepts. In this work, we address
  this shortcoming by proposing an approach to enable personalization
  capabilities in existing text-to-image diffusion models. We propose a novel
  architecture (BootPIG) that allows a user to provide reference images of an
  object in order to guide the appearance of a concept in the generated images.
    The proposed BootPIG architecture makes minimal modifications to a pretrained
  text-to-image diffusion model and utilizes a separate UNet model to steer the
  generations toward the desired appearance. We introduce a training procedure
  that allows us to bootstrap personalization capabilities in the BootPIG
  architecture using data generated from pretrained text-to-image models, LLM
  chat agents, and image segmentation models. In contrast to existing methods
  that require several days of pretraining, the BootPIG architecture can be
  trained in approximately 1 hour. Experiments on the DreamBooth dataset
  demonstrate that BootPIG outperforms existing zero-shot methods while being
  comparable with test-time finetuning approaches. Through a user study, we
  validate the preference for BootPIG generations over existing methods both in
  maintaining fidelity to the reference object's appearance and aligning with
  textual prompts.
paper_authors._id: 65b3149af32c79fea798e6ec
paper_authors.name: Senthil Purushwalkam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:17:43.719Z
paper_authors._id: 637e633366a51ca5d960a9b1
paper_authors.avatarUrl: /avatars/e53242e4e36875105be087c46a552e29.svg
paper_authors.fullname: Senthil Purushwalkam
paper_authors.type: user
paper_authors.user: senthilps8
paper_authors._id: 65b3149af32c79fea798e6ed
paper_authors.name: Akash Gokul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:17:49.780Z
paper_authors._id: 63093856b105f8675bd1c783
paper_authors.avatarUrl: /avatars/505a7b99893330a36a136d13fa99e05e.svg
paper_authors.fullname: Akash Gokul
paper_authors.type: user
paper_authors.user: akashgokul
paper_authors._id: 65b3149af32c79fea798e6ee
paper_authors.name: Shafiq Joty
paper_authors._id: 65b3149af32c79fea798e6ef
paper_authors.name: Nikhil Naik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:18:04.969Z
paper_authors._id: 627c2c751b02d7b5bf663fb0
paper_authors.avatarUrl: /avatars/f33ac735e2cc910d2da1286d476de5db.svg
paper_authors.fullname: Nikhil Naik
paper_authors.type: user
paper_authors.user: nikdnaik
paper_title: BootPIG: Bootstrapping Zero-shot Personalized Image Generation
    Capabilities in Pretrained Diffusion Models
paper_summary: Recent text-to-image generation models have demonstrated incredible success
  in generating images that faithfully follow input prompts. However, the
  requirement of using words to describe a desired concept provides limited
  control over the appearance of the generated concepts. In this work, we address
  this shortcoming by proposing an approach to enable personalization
  capabilities in existing text-to-image diffusion models. We propose a novel
  architecture (BootPIG) that allows a user to provide reference images of an
  object in order to guide the appearance of a concept in the generated images.
    The proposed BootPIG architecture makes minimal modifications to a pretrained
  text-to-image diffusion model and utilizes a separate UNet model to steer the
  generations toward the desired appearance. We introduce a training procedure
  that allows us to bootstrap personalization capabilities in the BootPIG
  architecture using data generated from pretrained text-to-image models, LLM
  chat agents, and image segmentation models. In contrast to existing methods
  that require several days of pretraining, the BootPIG architecture can be
  trained in approximately 1 hour. Experiments on the DreamBooth dataset
  demonstrate that BootPIG outperforms existing zero-shot methods while being
  comparable with test-time finetuning approaches. Through a user study, we
  validate the preference for BootPIG generations over existing methods both in
  maintaining fidelity to the reference object's appearance and aligning with
  textual prompts.
paper_ai_summary: BootPIG architecture enhances text-to-image generation models with personalization capabilities, using reference images, to improve the fidelity to the desired appearance while maintaining alignment with textual prompts.
paper_ai_keywords: text-to-image diffusion models BootPIG UNet DreamBooth zero-shot methods test-time finetuning