  As one of the most fundamental techniques in multimodal learning, cross-modal
matching aims to project various sensory modalities into a shared feature
space. To achieve this, massive and correctly aligned data pairs are required
for model training. However, unlike unimodal datasets, multimodal datasets are
extremely harder to collect and annotate precisely. As an alternative, the
co-occurred data pairs (e.g., image-text pairs) collected from the Internet
have been widely exploited in the area. Unfortunately, the cheaply collected
dataset unavoidably contains many mismatched data pairs, which have been proven
to be harmful to the model's performance. To address this, we propose a general
framework called BiCro (Bidirectional Cross-modal similarity consistency),
which can be easily integrated into existing cross-modal matching models and
improve their robustness against noisy data. Specifically, BiCro aims to
estimate soft labels for noisy data pairs to reflect their true correspondence
degree. The basic idea of BiCro is motivated by that -- taking image-text
matching as an example -- similar images should have similar textual
descriptions and vice versa. Then the consistency of these two similarities can
be recast as the estimated soft labels to train the matching model. The
experiments on three popular cross-modal matching datasets demonstrate that our
method significantly improves the noise-robustness of various matching models,
and surpass the state-of-the-art by a clear margin.

Computer Vision and Pattern Recognition