title: TextBoost: Towards One-Shot Personalization of Text-to-Image Models via
    Fine-tuning Text Encoder
summary: Recent breakthroughs in text-to-image models have opened up promising
  research avenues in personalized image generation, enabling users to create
  diverse images of a specific subject using natural language prompts. However,
  existing methods often suffer from performance degradation when given only a
  single reference image. They tend to overfit the input, producing highly
  similar outputs regardless of the text prompt. This paper addresses the
  challenge of one-shot personalization by mitigating overfitting, enabling the
  creation of controllable images through text prompts. Specifically, we propose
  a selective fine-tuning strategy that focuses on the text encoder. Furthermore,
  we introduce three key techniques to enhance personalization performance: (1)
  augmentation tokens to encourage feature disentanglement and alleviate
  overfitting, (2) a knowledge-preservation loss to reduce language drift and
  promote generalizability across diverse prompts, and (3) SNR-weighted sampling
  for efficient training. Extensive experiments demonstrate that our approach
  efficiently generates high-quality, diverse images using only a single
  reference image while significantly reducing memory and storage requirements.
paper_authors._id: 66e39aa29d477f566f5a4fd0
paper_authors.name: NaHyeon Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-13T12:09:02.925Z
paper_authors._id: 65df402343bf08064bd72dc2
paper_authors.avatarUrl: /avatars/e4d3f339e17564f7d8f67e665813938c.svg
paper_authors.fullname: NaHyeon Park
paper_authors.type: user
paper_authors.user: nahyeonkaty
paper_authors._id: 66e39aa29d477f566f5a4fd1
paper_authors.name: Kunhee Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:58:53.477Z
paper_authors._id: 6314968cd8dc27b2f7f1f3e2
paper_authors.avatarUrl: /avatars/3830bcff76b0408b1849fd9a44f7aedc.svg
paper_authors.fullname: Kunhee Kim
paper_authors.type: user
paper_authors.user: kunheekim
paper_authors._id: 66e39aa29d477f566f5a4fd2
paper_authors.name: Hyunjung Shim
paper_title: TextBoost: Towards One-Shot Personalization of Text-to-Image Models via
    Fine-tuning Text Encoder
paper_summary: Recent breakthroughs in text-to-image models have opened up promising
  research avenues in personalized image generation, enabling users to create
  diverse images of a specific subject using natural language prompts. However,
  existing methods often suffer from performance degradation when given only a
  single reference image. They tend to overfit the input, producing highly
  similar outputs regardless of the text prompt. This paper addresses the
  challenge of one-shot personalization by mitigating overfitting, enabling the
  creation of controllable images through text prompts. Specifically, we propose
  a selective fine-tuning strategy that focuses on the text encoder. Furthermore,
  we introduce three key techniques to enhance personalization performance: (1)
  augmentation tokens to encourage feature disentanglement and alleviate
  overfitting, (2) a knowledge-preservation loss to reduce language drift and
  promote generalizability across diverse prompts, and (3) SNR-weighted sampling
  for efficient training. Extensive experiments demonstrate that our approach
  efficiently generates high-quality, diverse images using only a single
  reference image while significantly reducing memory and storage requirements.
paper_ai_summary: Selective fine-tuning of text encoders, along with augmentation tokens, knowledge-preservation loss, and SNR-weighted sampling, enhances personalized image generation from single reference images.
paper_ai_keywords: text-to-image models personalized image generation text encoder selective fine-tuning augmentation tokens feature disentanglement knowledge-preservation loss language drift SNR-weighted sampling