  It is well known that human gaze carries significant information about visual
attention. However, there are three main difficulties in incorporating the gaze
data in an attention mechanism of deep neural networks: 1) the gaze fixation
points are likely to have measurement errors due to blinking and rapid eye
movements; 2) it is unclear when and how much the gaze data is correlated with
visual attention; and 3) gaze data is not always available in many real-world
situations. In this work, we introduce an effective probabilistic approach to
integrate human gaze into spatiotemporal attention for egocentric activity
recognition. Specifically, we represent the locations of gaze fixation points
as structured discrete latent variables to model their uncertainties. In
addition, we model the distribution of gaze fixations using a variational
method. The gaze distribution is learned during the training process so that
the ground-truth annotations of gaze locations are no longer needed in testing
situations since they are predicted from the learned gaze distribution. The
predicted gaze locations are used to provide informative attentional cues to
improve the recognition performance. Our method outperforms all the previous
state-of-the-art approaches on EGTEA, which is a large-scale dataset for
egocentric activity recognition provided with gaze measurements. We also
perform an ablation study and qualitative analysis to demonstrate that our
attention mechanism is effective.

Computer Vision and Pattern Recognition