  The development of transformer-based models has resulted in significant
advances in addressing various vision and NLP-based research challenges.
However, the progress made in transformer-based methods has not been
effectively applied to biosensing research. This paper presents a novel Unified
Biosensor-Vision Multi-modal Transformer-based (UBVMT) method to classify
emotions in an arousal-valence space by combining a 2D representation of an
ECG/PPG signal with the face information. To achieve this goal, we first
investigate and compare the unimodal emotion recognition performance of three
image-based representations of the ECG/PPG signal. We then present our UBVMT
network which is trained to perform emotion recognition by combining the 2D
image-based representation of the ECG/PPG signal and the facial expression
features. Our unified transformer model consists of homogeneous transformer
blocks that take as an input the 2D representation of the ECG/PPG signal and
the corresponding face frame for emotion representation learning with minimal
modality-specific design. Our UBVMT model is trained by reconstructing masked
patches of video frames and 2D images of ECG/PPG signals, and contrastive
modeling to align face and ECG/PPG data. Extensive experiments on the
MAHNOB-HCI and DEAP datasets show that our Unified UBVMT-based model produces
comparable results to the state-of-the-art techniques.

Computer Vision and Pattern Recognition;Artificial Intelligence