title: ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity
    Preserving
summary: Diffusion-based technologies have made significant strides, particularly in
  personalized and customized facialgeneration. However, existing methods face
  challenges in achieving high-fidelity and detailed identity (ID)consistency,
  primarily due to insufficient fine-grained control over facial areas and the
  lack of a comprehensive strategy for ID preservation by fully considering
  intricate facial details and the overall face. To address these limitations, we
  introduce ConsistentID, an innovative method crafted for
  diverseidentity-preserving portrait generation under fine-grained multimodal
  facial prompts, utilizing only a single reference image. ConsistentID comprises
  two key components: a multimodal facial prompt generator that combines facial
  features, corresponding facial descriptions and the overall facial context to
  enhance precision in facial details, and an ID-preservation network optimized
  through the facial attention localization strategy, aimed at preserving ID
  consistency in facial regions. Together, these components significantly enhance
  the accuracy of ID preservation by introducing fine-grained multimodal ID
  information from facial regions. To facilitate training of ConsistentID, we
  present a fine-grained portrait dataset, FGID, with over 500,000 facial images,
  offering greater diversity and comprehensiveness than existing public facial
  datasets. % such as LAION-Face, CelebA, FFHQ, and SFHQ. Experimental results
  substantiate that our ConsistentID achieves exceptional precision and diversity
  in personalized facial generation, surpassing existing methods in the MyStyle
  dataset. Furthermore, while ConsistentID introduces more multimodal ID
  information, it maintains a fast inference speed during generation.
paper_authors._id: 662b20e08bccd5f54428fcab
paper_authors.name: Jiehui Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T11:56:55.893Z
paper_authors._id: 64d1efef4a204a4d125fd4fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d1efef4a204a4d125fd4fc/mOjMzPUkAR-SmzYwofazX.jpeg
paper_authors.fullname: JackAILab
paper_authors.type: user
paper_authors.user: JackAILab
paper_authors._id: 662b20e08bccd5f54428fcac
paper_authors.name: Xiao Dong
paper_authors._id: 662b20e08bccd5f54428fcad
paper_authors.name: Wenhui Song
paper_authors._id: 662b20e08bccd5f54428fcae
paper_authors.name: Hanhui Li
paper_authors._id: 662b20e08bccd5f54428fcaf
paper_authors.name: Jun Zhou
paper_authors._id: 662b20e08bccd5f54428fcb0
paper_authors.name: Yuhao Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:22:45.382Z
paper_authors._id: 64b741376c169983c9804c21
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b741376c169983c9804c21/C1L8svFi3InAiCcSBOY8x.jpeg
paper_authors.fullname: Yuhao Cheng
paper_authors.type: user
paper_authors.user: Rovio
paper_authors._id: 662b20e08bccd5f54428fcb1
paper_authors.name: Shutao Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:23:08.528Z
paper_authors._id: 65e8a6cab406b73958a799ab
paper_authors.avatarUrl: /avatars/0af594af009066938409fc3246c0b058.svg
paper_authors.fullname: Liao Shutao
paper_authors.type: user
paper_authors.user: kaideee
paper_authors._id: 662b20e08bccd5f54428fcb2
paper_authors.name: Long Chen
paper_authors._id: 662b20e08bccd5f54428fcb3
paper_authors.name: Yiqiang Yan
paper_authors._id: 662b20e08bccd5f54428fcb4
paper_authors.name: Shengcai Liao
paper_authors._id: 662b20e08bccd5f54428fcb5
paper_authors.name: Xiaodan Liang
paper_title: ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity
    Preserving
paper_summary: Diffusion-based technologies have made significant strides, particularly in
  personalized and customized facialgeneration. However, existing methods face
  challenges in achieving high-fidelity and detailed identity (ID)consistency,
  primarily due to insufficient fine-grained control over facial areas and the
  lack of a comprehensive strategy for ID preservation by fully considering
  intricate facial details and the overall face. To address these limitations, we
  introduce ConsistentID, an innovative method crafted for
  diverseidentity-preserving portrait generation under fine-grained multimodal
  facial prompts, utilizing only a single reference image. ConsistentID comprises
  two key components: a multimodal facial prompt generator that combines facial
  features, corresponding facial descriptions and the overall facial context to
  enhance precision in facial details, and an ID-preservation network optimized
  through the facial attention localization strategy, aimed at preserving ID
  consistency in facial regions. Together, these components significantly enhance
  the accuracy of ID preservation by introducing fine-grained multimodal ID
  information from facial regions. To facilitate training of ConsistentID, we
  present a fine-grained portrait dataset, FGID, with over 500,000 facial images,
  offering greater diversity and comprehensiveness than existing public facial
  datasets. % such as LAION-Face, CelebA, FFHQ, and SFHQ. Experimental results
  substantiate that our ConsistentID achieves exceptional precision and diversity
  in personalized facial generation, surpassing existing methods in the MyStyle
  dataset. Furthermore, while ConsistentID introduces more multimodal ID
  information, it maintains a fast inference speed during generation.
paper_ai_keywords: diffusion-based technologies fine-grained control identity (ID) consistency facial generation fine-grained multimodal facial prompts reference image multimodal facial prompt generator ID-preservation network facial detection facial recognition FGID dataset MyStyle dataset fine-grained portrait dataset facial regions NCEN regions