Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation
<|reference_start|>Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation: Gestures are pivotal in enhancing co-speech communication. While recent works have mostly focused on point-level motion transformation or fully supervised motion representations through data-driven approaches, we explore the representation of gestures in co-speech, with a focus on self-supervised representation and pixel-level motion deviation, utilizing a diffusion model which incorporates latent motion features. Our approach leverages self-supervised deviation in latent representation to facilitate hand gestures generation, which are crucial for generating realistic gesture videos. Results of our first experiment demonstrate that our method enhances the quality of generated videos, with an improvement from 2.7 to 4.5% for FGD, DIV, and FVD, and 8.1% for PSNR, 2.5% for SSIM over the current state-of-the-art methods.<|reference_end|>
@article{yang2024self-supervised,
    title={Self-Supervised Learning of Deviation in Latent Representation for
  Co-speech Gesture Video Generation},
    author={Huan Yang, Jiahui Chen, Chaofan Ding, Runhua Shi, Siyu Xiong, Qingqi
  Hong, Xiaoqi Mo, Xinhan Di},
    journal={arXiv preprint arXiv:2409.17674},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.17674},
    primaryClass={cs.CV}
}