title: SPF-Portrait: Towards Pure Portrait Customization with Semantic
    Pollution-Free Fine-tuning
summary: Fine-tuning a pre-trained Text-to-Image (T2I) model on a tailored portrait
  dataset is the mainstream method for text-driven customization of portrait
  attributes. Due to Semantic Pollution during fine-tuning, existing methods
  struggle to maintain the original model's behavior and achieve incremental
  learning while customizing target attributes. To address this issue, we propose
  SPF-Portrait, a pioneering work to purely understand customized semantics while
  eliminating semantic pollution in text-driven portrait customization. In our
  SPF-Portrait, we propose a dual-path pipeline that introduces the original
  model as a reference for the conventional fine-tuning path. Through contrastive
  learning, we ensure adaptation to target attributes and purposefully align
  other unrelated attributes with the original portrait. We introduce a novel
  Semantic-Aware Fine Control Map, which represents the precise response regions
  of the target semantics, to spatially guide the alignment process between the
  contrastive paths. This alignment process not only effectively preserves the
  performance of the original model but also avoids over-alignment. Furthermore,
  we propose a novel response enhancement mechanism to reinforce the performance
  of target attributes, while mitigating representation discrepancy inherent in
  direct cross-modal supervision. Extensive experiments demonstrate that
  SPF-Portrait achieves state-of-the-art performance. Project webpage:
  https://spf-portrait.github.io/SPF-Portrait/
paper_authors._id: 67f3c4f40db583bb63e03152
paper_authors.name: Xiaole Xian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T16:22:43.251Z
paper_authors._id: 651e27b6f6663789c8e9c5f5
paper_authors.avatarUrl: /avatars/539498d74844aa08741d3fd63b0aa96f.svg
paper_authors.fullname: Xian Xiaole
paper_authors.type: user
paper_authors.user: connorxian
paper_authors._id: 67f3c4f40db583bb63e03153
paper_authors.name: Zhichao Liao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:52:45.276Z
paper_authors._id: 66eb9d5d7f17088991ecbae2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66eb9d5d7f17088991ecbae2/khI2KZ4ZGJ891iOf-O0BF.jpeg
paper_authors.fullname: Zhichao Liao
paper_authors.type: user
paper_authors.user: ChaosLiao
paper_authors._id: 67f3c4f40db583bb63e03154
paper_authors.name: Qingyu Li
paper_authors._id: 67f3c4f40db583bb63e03155
paper_authors.name: Wenyu Qin
paper_authors._id: 67f3c4f40db583bb63e03156
paper_authors.name: Pengfei Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-04T10:05:44.304Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 67f3c4f40db583bb63e03157
paper_authors.name: Weicheng Xie
paper_authors._id: 67f3c4f40db583bb63e03158
paper_authors.name: Long Zeng
paper_authors._id: 67f3c4f40db583bb63e03159
paper_authors.name: Linlin Shen
paper_authors._id: 67f3c4f40db583bb63e0315a
paper_authors.name: Pingfa Feng
paper_title: SPF-Portrait: Towards Pure Portrait Customization with Semantic
    Pollution-Free Fine-tuning
paper_summary: Fine-tuning a pre-trained Text-to-Image (T2I) model on a tailored portrait
  dataset is the mainstream method for text-driven customization of portrait
  attributes. Due to Semantic Pollution during fine-tuning, existing methods
  struggle to maintain the original model's behavior and achieve incremental
  learning while customizing target attributes. To address this issue, we propose
  SPF-Portrait, a pioneering work to purely understand customized semantics while
  eliminating semantic pollution in text-driven portrait customization. In our
  SPF-Portrait, we propose a dual-path pipeline that introduces the original
  model as a reference for the conventional fine-tuning path. Through contrastive
  learning, we ensure adaptation to target attributes and purposefully align
  other unrelated attributes with the original portrait. We introduce a novel
  Semantic-Aware Fine Control Map, which represents the precise response regions
  of the target semantics, to spatially guide the alignment process between the
  contrastive paths. This alignment process not only effectively preserves the
  performance of the original model but also avoids over-alignment. Furthermore,
  we propose a novel response enhancement mechanism to reinforce the performance
  of target attributes, while mitigating representation discrepancy inherent in
  direct cross-modal supervision. Extensive experiments demonstrate that
  SPF-Portrait achieves state-of-the-art performance. Project webpage:
  https://spf-portrait.github.io/SPF-Portrait/
paper_ai_summary: SPF-Portrait addresses semantic pollution during fine-tuning by using a dual-path pipeline with contrastive learning and a Semantic-Aware Fine Control Map for accurate spatial alignment, enhancing target attributes while preserving the original model's performance.
paper_ai_keywords: fine-tuning pre-trained model Text-to-Image (T2I) Semantic Pollution dual-path pipeline contrastive learning Semantic-Aware Fine Control Map spatial alignment over-alignment response enhancement mechanism cross-modal supervision