title: HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion
    Models
summary: This paper explores advancements in high-fidelity personalized image
  generation through the utilization of pre-trained text-to-image diffusion
  models. While previous approaches have made significant strides in generating
  versatile scenes based on text descriptions and a few input images, challenges
  persist in maintaining the subject fidelity within the generated images. In
  this work, we introduce an innovative algorithm named HiFi Tuner to enhance the
  appearance preservation of objects during personalized image generation. Our
  proposed method employs a parameter-efficient fine-tuning framework, comprising
  a denoising process and a pivotal inversion process. Key enhancements include
  the utilization of mask guidance, a novel parameter regularization technique,
  and the incorporation of step-wise subject representations to elevate the
  sample fidelity. Additionally, we propose a reference-guided generation
  approach that leverages the pivotal inversion of a reference image to mitigate
  unwanted subject variations and artifacts. We further extend our method to a
  novel image editing task: substituting the subject in an image through textual
  manipulations. Experimental evaluations conducted on the DreamBooth dataset
  using the Stable Diffusion model showcase promising results. Fine-tuning solely
  on textual embeddings improves CLIP-T score by 3.6 points and improves DINO
  score by 9.6 points over Textual Inversion. When fine-tuning all parameters,
  HiFi Tuner improves CLIP-T score by 1.2 points and improves DINO score by 1.2
  points over DreamBooth, establishing a new state of the art.
paper_authors._id: 656d35fc903e16e62b054129
paper_authors.name: Zhonghao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:13:43.415Z
paper_authors._id: 645dafaaa19f3e64bbf2d9a8
paper_authors.avatarUrl: /avatars/539d7ac00b65a8e58b8c1bfa040545ef.svg
paper_authors.fullname: Zhonghao Wang
paper_authors.type: user
paper_authors.user: Zhonghao
paper_authors._id: 656d35fc903e16e62b05412a
paper_authors.name: Wei Wei
paper_authors._id: 656d35fc903e16e62b05412b
paper_authors.name: Yang Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-08T08:49:26.160Z
paper_authors._id: 631ba6aaf6bc4be4a653b2c7
paper_authors.avatarUrl: /avatars/c026f47a042711127d3b8431f6217f55.svg
paper_authors.fullname: Yang Zhao
paper_authors.type: user
paper_authors.user: YangNaruto
paper_authors._id: 656d35fc903e16e62b05412c
paper_authors.name: Zhisheng Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:14:39.044Z
paper_authors._id: 630846e82d33960e6b7a43ce
paper_authors.avatarUrl: /avatars/058ad9a17d51ce9e1952d2677b0a47bc.svg
paper_authors.fullname: Xiao
paper_authors.type: user
paper_authors.user: Zhisheng
paper_authors._id: 656d35fc903e16e62b05412d
paper_authors.name: Mark Hasegawa-Johnson
paper_authors._id: 656d35fc903e16e62b05412e
paper_authors.name: Humphrey Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:14:15.345Z
paper_authors._id: 61e1188afc27c0f5e3641eb3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
paper_authors.fullname: Humphrey Shi
paper_authors.type: user
paper_authors.user: Humphrey
paper_authors._id: 656d35fc903e16e62b05412f
paper_authors.name: Tingbo Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:14:27.649Z
paper_authors._id: 655846d7ed8df83128f5826a
paper_authors.avatarUrl: /avatars/d7ce174d7d1b8614d5f6f071225c0057.svg
paper_authors.fullname: Hou
paper_authors.type: user
paper_authors.user: Tingbo
paper_title: HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion
    Models
paper_summary: This paper explores advancements in high-fidelity personalized image
  generation through the utilization of pre-trained text-to-image diffusion
  models. While previous approaches have made significant strides in generating
  versatile scenes based on text descriptions and a few input images, challenges
  persist in maintaining the subject fidelity within the generated images. In
  this work, we introduce an innovative algorithm named HiFi Tuner to enhance the
  appearance preservation of objects during personalized image generation. Our
  proposed method employs a parameter-efficient fine-tuning framework, comprising
  a denoising process and a pivotal inversion process. Key enhancements include
  the utilization of mask guidance, a novel parameter regularization technique,
  and the incorporation of step-wise subject representations to elevate the
  sample fidelity. Additionally, we propose a reference-guided generation
  approach that leverages the pivotal inversion of a reference image to mitigate
  unwanted subject variations and artifacts. We further extend our method to a
  novel image editing task: substituting the subject in an image through textual
  manipulations. Experimental evaluations conducted on the DreamBooth dataset
  using the Stable Diffusion model showcase promising results. Fine-tuning solely
  on textual embeddings improves CLIP-T score by 3.6 points and improves DINO
  score by 9.6 points over Textual Inversion. When fine-tuning all parameters,
  HiFi Tuner improves CLIP-T score by 1.2 points and improves DINO score by 1.2
  points over DreamBooth, establishing a new state of the art.
paper_ai_summary: HiFi Tuner enhances object appearance conservation in personalized image generation through mask guidance, parameter regularization, and step-wise subject representations, achieving state-of-the-art results on the DreamBooth dataset.
paper_ai_keywords: text-to-image diffusion models parameter-efficient fine-tuning denoising process pivotal inversion process mask guidance parameter regularization step-wise subject representations reference-guided generation CLIP-T score DINO score Textual Inversion DreamBooth dataset Stable Diffusion model