Movies and meaning: from low level features to mind reading
When dealing with movies, closing the tremendous discontinuity between low level features and the richness of semantics in the viewers' cognitive processes, requires a variety of approaches and different perspectives. For instance when attempting to relate movie content to users' affective responses, previous work suggests that a direct mapping of audio visual properties into elicited emotions is difficult, due to the high variability of individual reactions. To reduce the gap between the objective level of features and the subjective sphere of emotions, we exploit the intermediate representation of the connotative properties of movies: the set of shooting and editing conventions that help in transmitting meaning to the audience. One of these stylistic feature, the shot scale, i.e. the distance of the camera from the subject, effectively regulates theory of mind, indicating that increasing spatial proximity to the character triggers higher occurrence of mental state references in viewers' story descriptions. Movies are also becoming an important stimuli employed in neural decoding, an ambitious line of research within contemporary neuroscience aiming at "mindreading" In this field we address the challenge of producing decoding models for the reconstruction of perceptual contents by combining fMRI data and deep features in a hybrid model able to predict specific video object classes. Introduction Cinema is definable as the performing art based on the optical illusion of a moving image. Despite cinema movies owe its mass spread by invention of the Lumi ere brothers' cinematograph, humans have been always using the figurative arts to represent images of the world around them. Some studies attribute the cinema authorship even to Palaeolithic paintings [1] as they represent scenes of animals not in a statical pose but rather dynamically providing the illusion of moving objects, as in famous cave of Chauvet Pont d'Arc (see Figure 1) The 'scienti c' role of cinema Movies are not only one of the most known entertainment sources but have become, in the last half century, one of the preferred testbed addressed by an increasing number of scientific studies. This happens because, while watching movies, the viewer often becomes a participant, because his entire body his senses, his equilibrium, his imagination are all convinced that an imaginary event is really happening. Human intersubjectivity properties, such as empathy [2] or theory of mind [3] allow viewers to immerse themselves in the film and experience similar mental and physiological states of the real life [4] For this peculiarity, the analysis of movies and their content is of particular interest in different areas of science, beyond Cinematography itself, such as Psychology and Neuroscience, although with different purposes: cognitive film scholars carefully study various elements of the film viewing experience focusing on the mental activity of viewers as the main object of inquiry; psychologists describe the phenomenology of the film experience to study affective processes and human relationships; neuroscientists recently started using movie excerpts as controlled brain stimuli in brain imaging studies. The interest towards movie data of tech disciplines such as Computer Vision or Image Processing, traditionally confined to applications such as automatic content analysis or video compression, is nowadays expanding to the challenging aim of finding suitable representations of movie data useful for other disciplines to pursue their fundamental tasks. It is not a coincidence that sometimes the limit that separates these different areas of science becomes pretty blurry and that multidisciplinary approaches to the problems of semantics and cognition are seen as the most promising ones, where movies are often the common denominator that bring all these areas together. Methodologies and features Despite a heterogeneity of objectives, traditional approaches to semantic video analysis have been rather homogeneous, adopting a two stage architecture of feature extraction first, followed by a semantic interpretation, e.g. classification, regression, etc. In particular feature representations have been predominantly hand crafted, drawing upon significant domain knowledge from cinematographic theory and demanding a specialist effort to be translated into algorithms. Since few years however, we are witnessing a revolution in machine learning with the reinvigorated usage of neural networks in deep learning, which promises a solution to tasks that are easy Figure 1. Panel with horses in the cave Chauvet Pont d'Arc (Ard eche)
HVEI