MSSTNet: A Multi-Scale Spatio-Temporal CNN-Transformer Network for Dynamic Facial Expression Recognition
<|reference_start|>MSSTNet: A Multi-Scale Spatio-Temporal CNN-Transformer Network for Dynamic Facial Expression Recognition: Unlike typical video action recognition, Dynamic Facial Expression Recognition (DFER) does not involve distinct moving targets but relies on localized changes in facial muscles. Addressing this distinctive attribute, we propose a Multi-Scale Spatio-temporal CNN-Transformer network (MSSTNet). Our approach takes spatial features of different scales extracted by CNN and feeds them into a Multi-scale Embedding Layer (MELayer). The MELayer extracts multi-scale spatial information and encodes these features before sending them into a Temporal Transformer (T-Former). The T-Former simultaneously extracts temporal information while continually integrating multi-scale spatial information. This process culminates in the generation of multi-scale spatio-temporal features that are utilized for the final classification. Our method achieves state-of-the-art results on two in-the-wild datasets. Furthermore, a series of ablation experiments and visualizations provide further validation of our approach's proficiency in leveraging spatio-temporal information within DFER.<|reference_end|>
@article{wang2024msstnet:,
    title={MSSTNet: A Multi-Scale Spatio-Temporal CNN-Transformer Network for
  Dynamic Facial Expression Recognition},
    author={Linhuang Wang, Xin Kang, Fei Ding, Satoshi Nakagawa and Fuji Ren},
    journal={ICASSP 2024-2024 IEEE International Conference on Acoustics,
  Speech and Signal Processing (ICASSP). IEEE, 2024: 3015-3019},
    year={2024},
    doi={10.1109/ICASSP48485.2024.10446699},
    archivePrefix={arXiv},
    eprint={2404.08433},
    primaryClass={cs.CV}
}