KeySync: A Robust Approach for Leakage-free Lip Synchronization in High
  Resolution
Lip synchronization, known as the task of aligning lip movements in an
existing video with new input audio, is typically framed as a simpler variant
of audio-driven facial animation. However, as well as suffering from the usual
issues in talking head generation (e.g., temporal consistency), lip
synchronization presents significant new challenges such as expression leakage
from the input video and facial occlusions, which can severely impact
real-world applications like automated dubbing, but are often neglected in
existing works. To address these shortcomings, we present KeySync, a two-stage
framework that succeeds in solving the issue of temporal consistency, while
also incorporating solutions for leakage and occlusions using a carefully
designed masking strategy. We show that KeySync achieves state-of-the-art
results in lip reconstruction and cross-synchronization, improving visual
quality and reducing expression leakage according to LipLeak, our novel leakage
metric. Furthermore, we demonstrate the effectiveness of our new masking
approach in handling occlusions and validate our architectural choices through
several ablation studies. Code and model weights can be found at
https://antonibigata.github.io/KeySync.
68147d4d687b82a9b6308cfd
Antoni Bigata
admin_assigned
2025-05-05T07:45:57.696Z
640777812e309e65452491dd
/avatars/01b927675e78f212408168522f65fe36.svg
Antoni Bigata
user
toninio19
68147d4d687b82a9b6308cfe
Rodrigo Mira
68147d4d687b82a9b6308cff
Stella Bounareli
admin_assigned
2025-05-05T07:46:12.618Z
64523e727f7e72724b4d7572
/avatars/ba92447a1ab3f1f4ad9c42fe9c0210ac.svg
Stella Bounareli
user
StelaBou
68147d4d687b82a9b6308d00
Michał Stypułkowski
68147d4d687b82a9b6308d01
Konstantinos Vougioukas
68147d4d687b82a9b6308d02
Stavros Petridis
68147d4d687b82a9b6308d03
Maja Pantic
KeySync: A Robust Approach for Leakage-free Lip Synchronization in High
  Resolution
Lip synchronization, known as the task of aligning lip movements in an
existing video with new input audio, is typically framed as a simpler variant
of audio-driven facial animation. However, as well as suffering from the usual
issues in talking head generation (e.g., temporal consistency), lip
synchronization presents significant new challenges such as expression leakage
from the input video and facial occlusions, which can severely impact
real-world applications like automated dubbing, but are often neglected in
existing works. To address these shortcomings, we present KeySync, a two-stage
framework that succeeds in solving the issue of temporal consistency, while
also incorporating solutions for leakage and occlusions using a carefully
designed masking strategy. We show that KeySync achieves state-of-the-art
results in lip reconstruction and cross-synchronization, improving visual
quality and reducing expression leakage according to LipLeak, our novel leakage
metric. Furthermore, we demonstrate the effectiveness of our new masking
approach in handling occlusions and validate our architectural choices through
several ablation studies. Code and model weights can be found at
https://antonibigata.github.io/KeySync.
KeySync, a two-stage framework, addresses expression leakage and occlusions in lip synchronization, improving visual quality and achieving state-of-the-art results in lip reconstruction and cross-synchronization.
Lip synchronization audio-driven facial animation temporal consistency expression leakage facial occlusions KeySync LipLeak