According to the method, a time sequence deep convolutional neural network (TCN), a self-attention Transformer encoder and a multi-modal multi-head attention mechanism are adopted, and the invention relates to a model and a recognition method for estimating continuous-dimension emotions from multi-mode (auditory and visual) time sequence information. The method comprises the following steps: obtaining embedded feature expressions of different modalities for features input by different modalities; then, using the embedded feature expressions of different modes as input, and obtaining advanced representations of different modes by using a multi-mode Transform encoder; and finally, taking the high-level feature expressions of different modes as input, and calculating an emotional state value at each moment. According to the method, more attention is paid to the influence of some key moments on the current emotional state in the past in time sequence, interference caused by long-term emotional information is eliminated, and the robustness of the model is improved. Meanwhile, the accuracy of continuous dimension emotion estimation is obviously improved through a method of simultaneously correcting a time sequence context dependency relationship and a multi-modal interaction fusion relationship in the model.
Continuous dimension emotion recognition method based on Transform encoder and multi-head multi-modal attention