  We revisit the weakly supervised cross-modal face-name alignment task; that
is, given an image and a caption, we label the faces in the image with the
names occurring in the caption. Whereas past approaches have learned the latent
alignment between names and faces by uncertainty reasoning over a set of images
and their respective captions, in this paper, we rely on appropriate loss
functions to learn the alignments in a neural network setting and propose SECLA
and SECLA-B. SECLA is a Symmetry-Enhanced Contrastive Learning-based Alignment
model that can effectively maximize the similarity scores between corresponding
faces and names in a weakly supervised fashion. A variation of the model,
SECLA-B, learns to align names and faces as humans do, that is, learning from
easy to hard cases to further increase the performance of SECLA. More
specifically, SECLA-B applies a two-stage learning framework: (1) Training the
model on an easy subset with a few names and faces in each image-caption pair.
(2) Leveraging the known pairs of names and faces from the easy cases using a
bootstrapping strategy with additional loss to prevent forgetting and learning
new alignments at the same time. We achieve state-of-the-art results for both
the augmented Labeled Faces in the Wild dataset and the Celebrity Together
dataset. In addition, we believe that our methods can be adapted to other
multimodal news understanding tasks.

Computer Vision and Pattern Recognition