How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations
<|reference_start|>How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations: Facial expression perception in humans inherently relies on prior knowledge and contextual cues, contributing to efficient and flexible processing. For instance, multi-modal emotional context (such as voice color, affective text, body pose, etc.) can prompt people to perceive emotional expressions in objectively neutral faces. Drawing inspiration from this, we introduce a novel approach for facial expression classification that goes beyond simple classification tasks. Our model accurately classifies a perceived face and synthesizes the corresponding mental representation perceived by a human when observing a face in context. With this, our model offers visual insights into its internal decision-making process. We achieve this by learning two independent representations of content and context using a VAE-GAN architecture. Subsequently, we propose a novel attention mechanism for context-dependent feature adaptation. The adapted representation is used for classification and to generate a context-augmented expression. We evaluate synthesized expressions in a human study, showing that our model effectively produces approximations of human mental representations. We achieve State-of-the-Art classification accuracies of 81.01% on the RAVDESS dataset and 79.34% on the MEAD dataset. We make our code publicly available.<|reference_end|>
@article{blume2024how,
    title={How Do You Perceive My Face? Recognizing Facial Expressions in
  Multi-Modal Context by Modeling Mental Representations},
    author={Florian Blume, Runfeng Qu, Pia Bideau, Martin Maier, Rasha Abdel
  Rahman, Olaf Hellwich},
    journal={arXiv preprint arXiv:2409.02566},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.02566},
    primaryClass={cs.CV}
}