title: Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional
    Talking Portrait Generation
summary: Recent advances in Talking Head Generation (THG) have achieved impressive lip
  synchronization and visual quality through diffusion models; yet existing
  methods struggle to generate emotionally expressive portraits while preserving
  speaker identity. We identify three critical limitations in current emotional
  talking head generation: insufficient utilization of audio's inherent emotional
  cues, identity leakage in emotion representations, and isolated learning of
  emotion correlations. To address these challenges, we propose a novel framework
  dubbed as DICE-Talk, following the idea of disentangling identity with emotion,
  and then cooperating emotions with similar characteristics. First, we develop a
  disentangled emotion embedder that jointly models audio-visual emotional cues
  through cross-modal attention, representing emotions as identity-agnostic
  Gaussian distributions. Second, we introduce a correlation-enhanced emotion
  conditioning module with learnable Emotion Banks that explicitly capture
  inter-emotion relationships through vector quantization and attention-based
  feature aggregation. Third, we design an emotion discrimination objective that
  enforces affective consistency during the diffusion process through
  latent-space classification. Extensive experiments on MEAD and HDTF datasets
  demonstrate our method's superiority, outperforming state-of-the-art approaches
  in emotion accuracy while maintaining competitive lip-sync performance.
  Qualitative results and user studies further confirm our method's ability to
  generate identity-preserving portraits with rich, correlated emotional
  expressions that naturally adapt to unseen identities.
paper_authors._id: 6810746e4be021d4dcd8d4de
paper_authors.name: Weipeng Tan
paper_authors._id: 6810746e4be021d4dcd8d4df
paper_authors.name: Chuming Lin
paper_authors._id: 6810746e4be021d4dcd8d4e0
paper_authors.name: Chengming Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T09:56:46.843Z
paper_authors._id: 652fab9d04a34a9282bf29d6
paper_authors.avatarUrl: /avatars/cd5967b37ebb1225e9ae1d46f196e2e2.svg
paper_authors.fullname: Chengming Xu
paper_authors.type: user
paper_authors.user: ChengmingX
paper_authors._id: 6810746e4be021d4dcd8d4e1
paper_authors.name: FeiFan Xu
paper_authors._id: 6810746e4be021d4dcd8d4e2
paper_authors.name: Xiaobin Hu
paper_authors._id: 6810746e4be021d4dcd8d4e3
paper_authors.name: Xiaozhong Ji
paper_authors._id: 6810746e4be021d4dcd8d4e4
paper_authors.name: Junwei Zhu
paper_authors._id: 6810746e4be021d4dcd8d4e5
paper_authors.name: Chengjie Wang
paper_authors._id: 6810746e4be021d4dcd8d4e6
paper_authors.name: Yanwei Fu
paper_title: Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional
    Talking Portrait Generation
paper_summary: Recent advances in Talking Head Generation (THG) have achieved impressive lip
  synchronization and visual quality through diffusion models; yet existing
  methods struggle to generate emotionally expressive portraits while preserving
  speaker identity. We identify three critical limitations in current emotional
  talking head generation: insufficient utilization of audio's inherent emotional
  cues, identity leakage in emotion representations, and isolated learning of
  emotion correlations. To address these challenges, we propose a novel framework
  dubbed as DICE-Talk, following the idea of disentangling identity with emotion,
  and then cooperating emotions with similar characteristics. First, we develop a
  disentangled emotion embedder that jointly models audio-visual emotional cues
  through cross-modal attention, representing emotions as identity-agnostic
  Gaussian distributions. Second, we introduce a correlation-enhanced emotion
  conditioning module with learnable Emotion Banks that explicitly capture
  inter-emotion relationships through vector quantization and attention-based
  feature aggregation. Third, we design an emotion discrimination objective that
  enforces affective consistency during the diffusion process through
  latent-space classification. Extensive experiments on MEAD and HDTF datasets
  demonstrate our method's superiority, outperforming state-of-the-art approaches
  in emotion accuracy while maintaining competitive lip-sync performance.
  Qualitative results and user studies further confirm our method's ability to
  generate identity-preserving portraits with rich, correlated emotional
  expressions that naturally adapt to unseen identities.
paper_ai_summary: A novel framework, DICE-Talk, improves emotional expression in talking head generation by disentangling identity and emotion, capturing emotion correlations, and enforcing affective consistency through diffusion models.
paper_ai_keywords: diffusion models talking head generation lips synchronization visual quality audio-visual emotional cues cross-modal attention disentangled emotion embedder Gaussian distributions correlation-enhanced emotion conditioning module Emotion Banks vector quantization attention-based feature aggregation emotion discrimination objective latent-space classification