  Recognizing the human emotion automatically from visual characteristics plays
a vital role in many intelligent applications. Recently, gait-based emotion
recognition, especially gait skeletons-based characteristic, has attracted much
attention, while many available methods have been proposed gradually. The
popular pipeline is to first extract affective features from joint skeletons,
and then aggregate the skeleton joint and affective features as the feature
vector for classifying the emotion. However, the aggregation procedure of these
emerged methods might be rigid, resulting in insufficiently exploiting the
complementary relationship between skeleton joint and affective features.
Meanwhile, the long range dependencies in both spatial and temporal domains of
the gait sequence are scarcely considered. To address these issues, we propose
a novel two-stream network with transformer-based complementarity, termed as
TNTC. Skeleton joint and affective features are encoded into two individual
images as the inputs of two streams, respectively. A new transformer-based
complementarity module (TCM) is proposed to bridge the complementarity between
two streams hierarchically via capturing long range dependencies. Experimental
results demonstrate TNTC outperforms state-of-the-art methods on the latest
dataset in terms of accuracy.

Computer Vision and Pattern Recognition