  This paper revisits recognition of natural image pleasantness by employing
deep convolutional neural networks and affordable eye trackers. There exist
several approaches to recognize image pleasantness: (1) computer vision, and
(2) psychophysical signals. For natural images, computer vision approaches have
not been as successful as for abstract paintings and is lagging behind the
psychophysical signals like eye movements. Despite better results, the
scalability of eye movements is adversely affected by the sensor cost. While
the introduction of affordable sensors have helped the scalability issue by
making the sensors more accessible, the application of such sensors in a
loosely controlled human-computer interaction setup is not yet studied for
affective image tagging. On the other hand, deep convolutional neural networks
have boosted the performance of vision-based techniques significantly in recent
years. To investigate the current status in regard to affective image tagging,
we (1) introduce a new eye movement dataset using an affordable eye tracker,
(2) study the use of deep neural networks for pleasantness recognition, (3)
investigate the gap between deep features and eye movements. To meet these
ends, we record eye movements in a less controlled setup, akin to daily
human-computer interaction. We assess features from eye movements, visual
features, and their combination. Our results show that (1) recognizing natural
image pleasantness from eye movement under less restricted setup is difficult
and previously used techniques are prone to fail, and (2) visual class
categories are strong cues for predicting pleasantness, due to their
correlation with emotions, necessitating careful study of this phenomenon. This
latter finding is alerting as some deep learning approaches may fit to the
class category bias.

Computer Vision and Pattern Recognition