  In the field of multimodal sentiment analysis (MSA), a few studies have
leveraged the inherent modality correlation information stored in samples for
self-supervised learning. However, they feed the training pairs in a random
order without consideration of difficulty. Without human annotation, the
generated training pairs of self-supervised learning often contain noise. If
noisy or hard pairs are used for training at the easy stage, the model might be
stuck in bad local optimum. In this paper, we inject curriculum learning into
weakly supervised modality correlation learning. The weakly supervised
correlation learning leverages the label information to generate scores for
negative pairs to learn a more discriminative embedding space, where negative
pairs are defined as two unimodal embeddings from different samples. To assist
the correlation learning, we feed the training pairs to the model according to
difficulty by the proposed curriculum learning, which consists of elaborately
designed scoring and feeding functions. The scoring function computes the
difficulty of pairs using pre-trained and current correlation predictors, where
the pairs with large losses are defined as hard pairs. Notably, the hardest
pairs are discarded in our algorithm, which are assumed as noisy pairs.
Moreover, the feeding function takes the difference of correlation losses as
feedback to determine the feeding actions (`stay', `step back', or `step
forward'). The proposed method reaches state-of-the-art performance on MSA.

Machine Learning