  Visual Emotion Analysis (VEA) is attracting increasing attention. One of the
biggest challenges of VEA is to bridge the affective gap between visual clues
in a picture and the emotion expressed by the picture. As the granularity of
emotions increases, the affective gap increases as well. Existing deep
approaches try to bridge the gap by directly learning discrimination among
emotions globally in one shot without considering the hierarchical relationship
among emotions at different affective levels and the affective level of
emotions to be classified. In this paper, we present the Multi-level Dependent
Attention Network (MDAN) with two branches, to leverage the emotion hierarchy
and the correlation between different affective levels and semantic levels. The
bottom-up branch directly learns emotions at the highest affective level and
strictly follows the emotion hierarchy while predicting emotions at lower
affective levels. In contrast, the top-down branch attempt to disentangle the
affective gap by one-to-one mapping between semantic levels and affective
levels, namely, Affective Semantic Mapping. At each semantic level, a local
classifier learns discrimination among emotions at the corresponding affective
level. Finally, We integrate global learning and local learning into a unified
deep framework and optimize the network simultaneously. Moreover, to properly
extract and leverage channel dependencies and spatial attention while
disentangling the affective gap, we carefully designed two attention modules:
the Multi-head Cross Channel Attention module and the Level-dependent Class
Activation Map module. Finally, the proposed deep framework obtains new
state-of-the-art performance on six VEA benchmarks, where it outperforms
existing state-of-the-art methods by a large margin, e.g., +3.85% on the WEBEmo
dataset at 25 classes classification accuracy.

Computer Vision and Pattern Recognition