Deep self-supervised learning with visualisation for automatic gesture recognition
<|reference_start|>Deep self-supervised learning with visualisation for automatic gesture recognition: Gesture is an important mean of non-verbal communication, with visual modality allows human to convey information during interaction, facilitating peoples and human-machine interactions. However, it is considered difficult to automatically recognise gestures. In this work, we explore three different means to recognise hand signs using deep learning: supervised learning based methods, self-supervised methods and visualisation based techniques applied to 3D moving skeleton data. Self-supervised learning used to train fully connected, CNN and LSTM method. Then, reconstruction method is applied to unlabelled data in simulated settings using CNN as a backbone where we use the learnt features to perform the prediction in the remaining labelled data. Lastly, Grad-CAM is applied to discover the focus of the models. Our experiments results show that supervised learning method is capable to recognise gesture accurately, with self-supervised learning increasing the accuracy in simulated settings. Finally, Grad-CAM visualisation shows that indeed the models focus on relevant skeleton joints on the associated gesture.<|reference_end|>
@article{allemand2024deep,
    title={Deep self-supervised learning with visualisation for automatic gesture
  recognition},
    author={Fabien Allemand, Alessio Mazzela, Jun Villette, Decky Aspandi, Titus
  Zaharia},
    journal={arXiv preprint arXiv:2406.12440},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.12440},
    primaryClass={cs.CV cs.HC}
}