SelfSwapper: Self-Supervised Face Swapping via Shape Agnostic Masked AutoEncoder
<|reference_start|>SelfSwapper: Self-Supervised Face Swapping via Shape Agnostic Masked AutoEncoder: Face swapping has gained significant attention for its varied applications. Most previous face swapping approaches have relied on the seesaw game training scheme, also known as the target-oriented approach. However, this often leads to instability in model training and results in undesired samples with blended identities due to the target identity leakage problem. Source-oriented methods achieve more stable training with self-reconstruction objective but often fail to accurately reflect target image's skin color and illumination. This paper introduces the Shape Agnostic Masked AutoEncoder (SAMAE) training scheme, a novel self-supervised approach that combines the strengths of both target-oriented and source-oriented approaches. Our training scheme addresses the limitations of traditional training methods by circumventing the conventional seesaw game and introducing clear ground truth through its self-reconstruction training regime. Our model effectively mitigates identity leakage and reflects target albedo and illumination through learned disentangled identity and non-identity features. Additionally, we closely tackle the shape misalignment and volume discrepancy problems with new techniques, including perforation confusion and random mesh scaling. SAMAE establishes a new state-of-the-art, surpassing other baseline methods, preserving both identity and non-identity attributes without sacrificing on either aspect.<|reference_end|>
@article{lee2024selfswapper:,
    title={SelfSwapper: Self-Supervised Face Swapping via Shape Agnostic Masked
  AutoEncoder},
    author={Jaeseong Lee, Junha Hyung, Sohyun Jeong, Jaegul Choo},
    journal={arXiv preprint arXiv:2402.07370},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.07370},
    primaryClass={cs.CV cs.AI}
}