TextBoost: Towards One-Shot Personalization of Text-to-Image Models via
  Fine-tuning Text Encoder
Recent breakthroughs in text-to-image models have opened up promising
research avenues in personalized image generation, enabling users to create
diverse images of a specific subject using natural language prompts. However,
existing methods often suffer from performance degradation when given only a
single reference image. They tend to overfit the input, producing highly
similar outputs regardless of the text prompt. This paper addresses the
challenge of one-shot personalization by mitigating overfitting, enabling the
creation of controllable images through text prompts. Specifically, we propose
a selective fine-tuning strategy that focuses on the text encoder. Furthermore,
we introduce three key techniques to enhance personalization performance: (1)
augmentation tokens to encourage feature disentanglement and alleviate
overfitting, (2) a knowledge-preservation loss to reduce language drift and
promote generalizability across diverse prompts, and (3) SNR-weighted sampling
for efficient training. Extensive experiments demonstrate that our approach
efficiently generates high-quality, diverse images using only a single
reference image while significantly reducing memory and storage requirements.
66e39aa29d477f566f5a4fd0
NaHyeon Park
claimed_verified
2024-09-13T12:09:02.925Z
65df402343bf08064bd72dc2
/avatars/e4d3f339e17564f7d8f67e665813938c.svg
NaHyeon Park
user
nahyeonkaty
66e39aa29d477f566f5a4fd1
Kunhee Kim
admin_assigned
2024-09-13T07:58:53.477Z
6314968cd8dc27b2f7f1f3e2
/avatars/3830bcff76b0408b1849fd9a44f7aedc.svg
Kunhee Kim
user
kunheekim
66e39aa29d477f566f5a4fd2
Hyunjung Shim
TextBoost: Towards One-Shot Personalization of Text-to-Image Models via
  Fine-tuning Text Encoder
Recent breakthroughs in text-to-image models have opened up promising
research avenues in personalized image generation, enabling users to create
diverse images of a specific subject using natural language prompts. However,
existing methods often suffer from performance degradation when given only a
single reference image. They tend to overfit the input, producing highly
similar outputs regardless of the text prompt. This paper addresses the
challenge of one-shot personalization by mitigating overfitting, enabling the
creation of controllable images through text prompts. Specifically, we propose
a selective fine-tuning strategy that focuses on the text encoder. Furthermore,
we introduce three key techniques to enhance personalization performance: (1)
augmentation tokens to encourage feature disentanglement and alleviate
overfitting, (2) a knowledge-preservation loss to reduce language drift and
promote generalizability across diverse prompts, and (3) SNR-weighted sampling
for efficient training. Extensive experiments demonstrate that our approach
efficiently generates high-quality, diverse images using only a single
reference image while significantly reducing memory and storage requirements.
Selective fine-tuning of text encoders, along with augmentation tokens, knowledge-preservation loss, and SNR-weighted sampling, enhances personalized image generation from single reference images.
text-to-image models personalized image generation text encoder selective fine-tuning augmentation tokens feature disentanglement knowledge-preservation loss language drift SNR-weighted sampling