  Capturing the 'mutual gaze' of people is essential for understanding and
interpreting the social interactions between them. To this end, this paper
addresses the problem of detecting people Looking At Each Other (LAEO) in video
sequences. For this purpose, we propose LAEO-Net++, a new deep CNN for
determining LAEO in videos. In contrast to previous works, LAEO-Net++ takes
spatio-temporal tracks as input and reasons about the whole track. It consists
of three branches, one for each character's tracked head and one for their
relative position. Moreover, we introduce two new LAEO datasets: UCO-LAEO and
AVA-LAEO. A thorough experimental evaluation demonstrates the ability of
LAEO-Net++ to successfully determine if two people are LAEO and the temporal
window where it happens. Our model achieves state-of-the-art results on the
existing TVHID-LAEO video dataset, significantly outperforming previous
approaches. Finally, we apply LAEO-Net++ to a social network, where we
automatically infer the social relationship between pairs of people based on
the frequency and duration that they LAEO, and show that LAEO can be a useful
tool for guided search of human interactions in videos. The code is available
at https://github.com/AVAuco/laeonetplus.

Computer Vision and Pattern Recognition