HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion
  Models
This paper explores advancements in high-fidelity personalized image
generation through the utilization of pre-trained text-to-image diffusion
models. While previous approaches have made significant strides in generating
versatile scenes based on text descriptions and a few input images, challenges
persist in maintaining the subject fidelity within the generated images. In
this work, we introduce an innovative algorithm named HiFi Tuner to enhance the
appearance preservation of objects during personalized image generation. Our
proposed method employs a parameter-efficient fine-tuning framework, comprising
a denoising process and a pivotal inversion process. Key enhancements include
the utilization of mask guidance, a novel parameter regularization technique,
and the incorporation of step-wise subject representations to elevate the
sample fidelity. Additionally, we propose a reference-guided generation
approach that leverages the pivotal inversion of a reference image to mitigate
unwanted subject variations and artifacts. We further extend our method to a
novel image editing task: substituting the subject in an image through textual
manipulations. Experimental evaluations conducted on the DreamBooth dataset
using the Stable Diffusion model showcase promising results. Fine-tuning solely
on textual embeddings improves CLIP-T score by 3.6 points and improves DINO
score by 9.6 points over Textual Inversion. When fine-tuning all parameters,
HiFi Tuner improves CLIP-T score by 1.2 points and improves DINO score by 1.2
points over DreamBooth, establishing a new state of the art.
656d35fc903e16e62b054129
Zhonghao Wang
admin_assigned
2023-12-04T10:13:43.415Z
645dafaaa19f3e64bbf2d9a8
/avatars/539d7ac00b65a8e58b8c1bfa040545ef.svg
Zhonghao Wang
user
Zhonghao
656d35fc903e16e62b05412a
Wei Wei
656d35fc903e16e62b05412b
Yang Zhao
claimed_verified
2024-01-08T08:49:26.160Z
631ba6aaf6bc4be4a653b2c7
/avatars/c026f47a042711127d3b8431f6217f55.svg
Yang Zhao
user
YangNaruto
656d35fc903e16e62b05412c
Zhisheng Xiao
admin_assigned
2023-12-04T10:14:39.044Z
630846e82d33960e6b7a43ce
/avatars/058ad9a17d51ce9e1952d2677b0a47bc.svg
Xiao
user
Zhisheng
656d35fc903e16e62b05412d
Mark Hasegawa-Johnson
656d35fc903e16e62b05412e
Humphrey Shi
admin_assigned
2023-12-04T10:14:15.345Z
61e1188afc27c0f5e3641eb3
https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
Humphrey Shi
user
Humphrey
656d35fc903e16e62b05412f
Tingbo Hou
admin_assigned
2023-12-04T10:14:27.649Z
655846d7ed8df83128f5826a
/avatars/d7ce174d7d1b8614d5f6f071225c0057.svg
Hou
user
Tingbo
HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion
  Models
This paper explores advancements in high-fidelity personalized image
generation through the utilization of pre-trained text-to-image diffusion
models. While previous approaches have made significant strides in generating
versatile scenes based on text descriptions and a few input images, challenges
persist in maintaining the subject fidelity within the generated images. In
this work, we introduce an innovative algorithm named HiFi Tuner to enhance the
appearance preservation of objects during personalized image generation. Our
proposed method employs a parameter-efficient fine-tuning framework, comprising
a denoising process and a pivotal inversion process. Key enhancements include
the utilization of mask guidance, a novel parameter regularization technique,
and the incorporation of step-wise subject representations to elevate the
sample fidelity. Additionally, we propose a reference-guided generation
approach that leverages the pivotal inversion of a reference image to mitigate
unwanted subject variations and artifacts. We further extend our method to a
novel image editing task: substituting the subject in an image through textual
manipulations. Experimental evaluations conducted on the DreamBooth dataset
using the Stable Diffusion model showcase promising results. Fine-tuning solely
on textual embeddings improves CLIP-T score by 3.6 points and improves DINO
score by 9.6 points over Textual Inversion. When fine-tuning all parameters,
HiFi Tuner improves CLIP-T score by 1.2 points and improves DINO score by 1.2
points over DreamBooth, establishing a new state of the art.
HiFi Tuner enhances object appearance conservation in personalized image generation through mask guidance, parameter regularization, and step-wise subject representations, achieving state-of-the-art results on the DreamBooth dataset.
text-to-image diffusion models parameter-efficient fine-tuning denoising process pivotal inversion process mask guidance parameter regularization step-wise subject representations reference-guided generation CLIP-T score DINO score Textual Inversion DreamBooth dataset Stable Diffusion model