AUD-TGN: Advancing Action Unit Detection with Temporal Convolution and GPT-2 in Wild Audiovisual Contexts
<|reference_start|>AUD-TGN: Advancing Action Unit Detection with Temporal Convolution and GPT-2 in Wild Audiovisual Contexts: Leveraging the synergy of both audio data and visual data is essential for understanding human emotions and behaviors, especially in in-the-wild setting. Traditional methods for integrating such multimodal information often stumble, leading to less-than-ideal outcomes in the task of facial action unit detection. To overcome these shortcomings, we propose a novel approach utilizing audio-visual multimodal data. This method enhances audio feature extraction by leveraging Mel Frequency Cepstral Coefficients (MFCC) and Log-Mel spectrogram features alongside a pre-trained VGGish network. Moreover, this paper adaptively captures fusion features across modalities by modeling the temporal relationships, and ultilizes a pre-trained GPT-2 model for sophisticated context-aware fusion of multimodal information. Our method notably improves the accuracy of AU detection by understanding the temporal and contextual nuances of the data, showcasing significant advancements in the comprehension of intricate scenarios. These findings underscore the potential of integrating temporal dynamics and contextual interpretation, paving the way for future research endeavors.<|reference_end|>
@article{yu2024aud-tgn:,
    title={AUD-TGN: Advancing Action Unit Detection with Temporal Convolution and
  GPT-2 in Wild Audiovisual Contexts},
    author={Jun Yu and Zerui Zhang and Zhihong Wei and Gongpeng Zhao and Zhongpeng
  Cai and Yongqi Wang and Guochen Xie and Jichao Zhu and Wangyuan Zhu},
    journal={arXiv preprint arXiv:2403.13678},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.13678},
    primaryClass={cs.CV}
}