title: KeySync: A Robust Approach for Leakage-free Lip Synchronization in High
    Resolution
summary: Lip synchronization, known as the task of aligning lip movements in an
  existing video with new input audio, is typically framed as a simpler variant
  of audio-driven facial animation. However, as well as suffering from the usual
  issues in talking head generation (e.g., temporal consistency), lip
  synchronization presents significant new challenges such as expression leakage
  from the input video and facial occlusions, which can severely impact
  real-world applications like automated dubbing, but are often neglected in
  existing works. To address these shortcomings, we present KeySync, a two-stage
  framework that succeeds in solving the issue of temporal consistency, while
  also incorporating solutions for leakage and occlusions using a carefully
  designed masking strategy. We show that KeySync achieves state-of-the-art
  results in lip reconstruction and cross-synchronization, improving visual
  quality and reducing expression leakage according to LipLeak, our novel leakage
  metric. Furthermore, we demonstrate the effectiveness of our new masking
  approach in handling occlusions and validate our architectural choices through
  several ablation studies. Code and model weights can be found at
  https://antonibigata.github.io/KeySync.
paper_authors._id: 68147d4d687b82a9b6308cfd
paper_authors.name: Antoni Bigata
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:45:57.696Z
paper_authors._id: 640777812e309e65452491dd
paper_authors.avatarUrl: /avatars/01b927675e78f212408168522f65fe36.svg
paper_authors.fullname: Antoni Bigata
paper_authors.type: user
paper_authors.user: toninio19
paper_authors._id: 68147d4d687b82a9b6308cfe
paper_authors.name: Rodrigo Mira
paper_authors._id: 68147d4d687b82a9b6308cff
paper_authors.name: Stella Bounareli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:46:12.618Z
paper_authors._id: 64523e727f7e72724b4d7572
paper_authors.avatarUrl: /avatars/ba92447a1ab3f1f4ad9c42fe9c0210ac.svg
paper_authors.fullname: Stella Bounareli
paper_authors.type: user
paper_authors.user: StelaBou
paper_authors._id: 68147d4d687b82a9b6308d00
paper_authors.name: Michał Stypułkowski
paper_authors._id: 68147d4d687b82a9b6308d01
paper_authors.name: Konstantinos Vougioukas
paper_authors._id: 68147d4d687b82a9b6308d02
paper_authors.name: Stavros Petridis
paper_authors._id: 68147d4d687b82a9b6308d03
paper_authors.name: Maja Pantic
paper_title: KeySync: A Robust Approach for Leakage-free Lip Synchronization in High
    Resolution
paper_summary: Lip synchronization, known as the task of aligning lip movements in an
  existing video with new input audio, is typically framed as a simpler variant
  of audio-driven facial animation. However, as well as suffering from the usual
  issues in talking head generation (e.g., temporal consistency), lip
  synchronization presents significant new challenges such as expression leakage
  from the input video and facial occlusions, which can severely impact
  real-world applications like automated dubbing, but are often neglected in
  existing works. To address these shortcomings, we present KeySync, a two-stage
  framework that succeeds in solving the issue of temporal consistency, while
  also incorporating solutions for leakage and occlusions using a carefully
  designed masking strategy. We show that KeySync achieves state-of-the-art
  results in lip reconstruction and cross-synchronization, improving visual
  quality and reducing expression leakage according to LipLeak, our novel leakage
  metric. Furthermore, we demonstrate the effectiveness of our new masking
  approach in handling occlusions and validate our architectural choices through
  several ablation studies. Code and model weights can be found at
  https://antonibigata.github.io/KeySync.
paper_ai_summary: KeySync, a two-stage framework, addresses expression leakage and occlusions in lip synchronization, improving visual quality and achieving state-of-the-art results in lip reconstruction and cross-synchronization.
paper_ai_keywords: Lip synchronization audio-driven facial animation temporal consistency expression leakage facial occlusions KeySync LipLeak