Computational mechanisms underlying cortical responses to the affordance properties of visual scenes
Recent advances in the use of deep neural networks for computer vision have yielded image computable models that exhibit human-level performance on scene- and object-classification tasks 1–4 ., The units in these networks often exhibit response profiles that are predictive of neural activity in mammalian visual cortex 5–11 , suggesting that they might be profitably used to investigate the computational algorithms that underlie biological vision 12–16 ., However , many of the internal operations of these models remain mysterious , and the fundamental theoretical principles that account for their predictive accuracy are not well understood 16–18 ., This presents an important challenge to the field: if deep neural networks are to fulfill their potential as a method for investigating visual perception in living organisms , it will first be necessary to develop techniques for using these networks to provide computational insights into neurobiological systems ., It is this issue—the use of deep neural networks for gaining insights into the computational processes of biological vision—that we address here ., We focus in particular on the mechanisms underlying natural scene perception ., A central aspect of scene perception is the identification of the navigational affordances of the local environment—where one can move to ( e . g . , a doorway or an unobstructed path ) , and where ones movement is blocked ., In a recent fMRI study , we showed that the navigational-affordance structure of scenes could be decoded from multivoxel response patterns in scene-selective visual areas 19 ., The strongest results were found in a region of the dorsal occipital lobe known as the occipital place area ( OPA ) , which is one of three patches of high-level visual cortex that respond strongly and preferentially to images of spatial scenes 20–24 ., These results demonstrated that the OPA encodes affordance-related visual features ., However , they did not address the crucial question of how these features might be computed from sensory inputs ., There was one aspect of the previous study that provided a clue as to how affordance representations might be constructed: affordance information was present in the OPA even though participants performed tasks that made no explicit reference to this information ., For example , in one experiment , participants were simply asked to report the colors of dots overlaid on the scene , and in another experiment , they were asked to perform a category-recognition task ., Despite the fact that these tasks did not require the participants to think about the spatial layout of the scene or plan a route through it , it was possible to decode navigational affordances in the OPA in both cases ., This suggested to us that affordances might be rapidly and automatically extracted through a set of purely feedforward computations ., In the current study we tested this idea by examining a biologically inspired CNN with a feedforward architecture that was previously trained for scene classification 3 ., This CNN implements a hierarchy of linear-nonlinear operations that give rise to increasingly complex feature representations , and previous work has shown that its internal representations can be used to predict neural responses to natural scene images 25 , 26 ., It has also been shown that the higher layers of this CNN can be used to decode the coarse spatial properties of scenes , such as their overall size 25 ., By examining this CNN , we aimed to demonstrate that affordance information could be extracted by a feedforward system , and to better understand how this information might be computed ., To preview our results , we find that the CNN contains information about fine-grained spatial features that could be used to map out the navigational pathways within a scene; moreover , these features are highly predictive of affordance-related fMRI responses in the OPA ., These findings demonstrate that the CNN can serve as a candidate , image-computable model of navigational-affordance coding in the human visual system ., Using this quantitative model , we then develop a set of techniques that provide insights into the computational operations that give rise to affordance-related representations ., These analyses reveal a set of stimulus input features that are critical for predicting affordance-related cortical responses , and they suggest a set of high-level , complex features that may serve as a basis set for the population coding of navigational affordances ., By combining neuroimaging findings with a fully quantitative computational model , we were able to complement a theory of cortical representation with discoveries of its algorithmic implementation—thus providing insights at multiple levels of understanding and moving us toward a more comprehensive functional description of visual cortex ., To test for the representation of navigational affordances in the human visual system , we examined fMRI responses to 50 images of indoor environments with clear navigational paths passing through the bottom of the scene ( Fig 1A ) ., Subjects viewed these images one at a time for 1 . 5 s each while maintaining central fixation and performing a category-recognition task that was unrelated to navigation ( i . e . , press a button when the viewed scene was a bathroom ) ., Details of the experimental paradigm and a complete analysis of the fMRI responses can be found in a previous report 19 ., In this section , we briefly recapitulate the aspects of the results that are most relevant to the subsequent computational analyses ., To measure the navigational affordances of these stimuli , we asked an independent group of subjects to indicate with a computer mouse the paths that they would take to walk through each environment starting from the bottom of the image ( Fig 1B ) ., From these responses , we created probabilistic maps of the navigational paths through each scene ., We then constructed histograms of these navigational probability measurements in one-degree angular bins over a range of directions radiating from the starting point of the paths ., These histograms approximate a probabilistic affordance map of potential navigational paths radiating from the perspective of the viewer 27 ., We then tested for the presence of affordance-related information in fMRI responses using representational similarity analysis ( RSA ) 28 ., In RSA , the information encoded in brain responses is compared with a cognitive or computational model through correlations of their representational dissimilarity matrices ( RDMs ) ., RDMs are constructed through pairwise comparisons of the model representations or brain responses for all stimulus classes ( in this case , the 50 images ) , and they serve as a summary measurement of the stimulus-class distinctions ., The correlation between any two RDMs reflects the degree to which they contain similar information about the stimuli ., We constructed an RDM for the navigational-affordance model through pairwise comparisons of the affordance histograms ( Fig 1C ) ., Neural RDMs were constructed for several regions of interest ( ROIs ) through pairwise comparisons of their multivoxel activation patterns for each image ., We focused our initial analyses on three ROIs that are known to be strongly involved in scene processing: the OPA , the parahippocampal place area ( PPA ) , and the retrosplenial complex ( RSC ) 20–24 ., All three of these regions respond more strongly to spatial scenes ( e . g . , images of landscapes , city streets , or rooms ) than other visual stimuli , such as objects and faces , and thus are good candidates for supporting representations of navigational affordances ., We also examined patterns in early visual cortex ( EVC ) ., Using RSA to compare the RDMs for these regions to the navigational-affordance RDM , we found evidence that affordance information is encoded in scene-selective visual cortex , most strongly in the dorsal scene-selective region known as the OPA ( Fig 1C ) ., These effects were not observed in lower-level EVC , suggesting that navigational affordances likely reflect mid-to-high-level visual features that require several computational stages along the cortical hierarchy ., In our previous report , a whole-brain searchlight analysis confirmed that the strongest cortical locus of affordance coding overlapped with the OPA 19 ., Interestingly , affordance coding in scene regions was observed even though participants performed a perceptual-semantic recognition task in which they were not explicitly asked about the navigational affordances of the scene—suggesting that affordance information is automatically elicited during scene perception ., Together , these results suggest that scene-selective visual cortex routinely encodes complex spatial features that can be used to map out the navigational affordances of the local visual scene ., These analyses provide functional insights into visual cortex at the level of representation—that is , the identification of sensory information encoded in cortical responses ., However , an equally important question for any theory of sensory cortical function is to understand how its representations can be computed at an algorithmic level 12–16 ., Understanding the algorithms that give rise to high-level sensory representations requires a quantitative model that implements representational transformations from visual stimuli ., Thus , we next turn to the question of how affordance representations might be computed from sensory inputs ., Visual cortex implements a complex set of highly nonlinear transformations that remain poorly understood ., Attempts at modeling these transformations using hand-engineered algorithms have long fallen short of accurately predicting mid-to-high-level sensory representations 6 , 10 , 11 , 29–31 ., However , advances in the development of artificial deep neural networks have dramatically changed the outlook for the quantitative modeling of visual cortex ., In particular , recently developed deep CNNs for tasks such as image classification have been found to predict sensory responses throughout much of visual cortex at an unprecedented level of accuracy 5–11 ., The performance of these CNNs suggests that they hold the promise of providing fundamental insights into the computational algorithms of biological vision ., However , because their internal representations were not hand-engineered to test specific theoretical operations , they are challenging to interpret ., Indeed , most of the critical parameters in CNNs are set through supervised learning for the purpose of achieving accurate performance on computer vision tasks , meaning that the resulting features are unconstrained by a priori theoretical principles ., Furthermore , the complex transformations of these internal CNN units cannot be understood through a simple inspection of their learned parameters ., Thus , neural network models have the potential to be highly informative to sensory neuroscience , but a critical challenge for moving forward is the development of techniques to probe the factors that best account for similarities between cortical responses and the internal representations of the models ., Here we tested a deep CNN as a potential candidate model of affordance-related responses in scene-selective visual cortex ., Given the apparent automaticity of affordance-related responses , we hypothesized that they could be modeled through a set of purely feedforward computations performed on image inputs ., To test this idea , we examined a model that was previously trained to classify images into a set of scene categories 3 ., This feedforward model contains 5 convolutional layers followed by 3 fully connected layers , the last of which contains units corresponding to a set of scene category labels ( Fig 2A ) ., The architecture of the model is similar to the AlexNet model that initiated the recent surge of interest in CNNs for computer vision 2 ., Units in the convolutional layers of this model have local connectivity , giving rise to increasingly large spatial receptive fields from layers 1 through 5 ., The dense connectivity of the final three layers means that the selectivity of their units could depend on any spatial position in the image ., Each unit in the CNN implements a linear-nonlinear operation in which it computes a weighted linear sum of its inputs followed by a nonlinear activation function ( specifically , a rectified linear threshold ) ., The weights on the inputs for each unit define a type of filter , and each convolutional layer contains a set of filters that are replicated with the same set of weights over all parts of the image ( hence the term “convolution” ) ., There are two other nonlinear operations implemented by a subset of the convolutional layers: max-pooling , in which only the maximum activation in a local pool of units is passed to the next layer , and normalization , in which activations are adjusted through division by a factor that reflects the summed activity of multiple units at the same spatial position ., Together , this small set of functional operations along with a set of architectural constraints define an untrained model whose many other parameters can be set through gradient descent with backpropagation—producing a trained model that performs highly complex feats of visual classification ., We passed the images from the fMRI experiment through the CNN and constructed a set of RDMs using the final outputs from each layer ., We then used RSA to compare the representations of the CNN with:, ( i ) the RDM for the navigational-affordance model and, ( ii ) the RDM for fMRI responses in the OPA ., The RSA comparisons with the affordance model showed that the CNN contained affordance-related information , which arose gradually across the lower layers and peaked in layer 5 , the highest convolutional layer ( Fig 2B ) ., Note that this was the case despite the fact that the CNN was trained to classify scenes based on their categorical identity ( e . g . , kitchen ) , not their affordance structure ., Weak effects were observed in lower convolutional layers , consistent with the pattern of findings from the fMRI experiment , in which affordance representations were not evident in EVC , and they suggest that affordances reflect mid-to-high-level , rather than low-level , visual features ., The decrease in affordance-related information in the last three fully connected layers may result from the increasingly semantic nature of representations in these layers , which ultimately encode a set of scene-category labels that are likely unrelated to the affordance-related features of the scenes ., The RSA comparisons with OPA responses showed that the CNN provided a highly accurate model of representations in this brain region , with strong effects across all CNN layers and a peak correlation in layer 5 ( Fig 2B ) ., Indeed , several layers of the CNN reached the highest accuracy we could expect for any model , given the noise ceiling of the OPA , which was calculated from the variance across subjects ( r-value for OPA noise ceiling = 0 . 30 ) ., Together , these findings demonstrate the feasibility of computing complex affordance-related features through a set of purely feedforward transformations , and they show that the CNN is a highly predictive model of OPA responses to natural images depicting such affordances ., The above findings demonstrate that the CNN is representationally similar to the navigational-affordance RDM and also similar to the OPA RDM , but they leave open the important question of whether the CNN captures the same variance in the OPA as the navigational-affordance RDM ., In other words , can the CNN serve as a computational model for affordance-related responses in the OPA ?, To address this question , we combined the RSA approach with commonality analysis 32 , a variance partitioning technique in which the explained variance of a multiple regression model is divided into the unique and shared variance contributed by all of its predictors ., In this case , multiple regression RSA was used to construct an encoding model of OPA representations ., Thus , the OPA was the predictand and the affordance and CNN models were predictors ., Our goal was to identify the portion of the shared variance between the affordance RDM and OPA RDM that could be accounted for by the CNN RDM ( Fig 3A ) ., This analysis showed that the CNN could explain a substantial portion of the representational similarity between the navigational-affordance model and the OPA ., In particular , over half of the explained variance of the navigational-affordance RDM could be accounted for by layer 5 of the CNN ( Fig 3B ) ., This suggests that the CNN can serve as a candidate , quantitative model of affordance-related responses in the OPA ., One of the most important aspects of the CNN as a candidate model of affordance-related cortical responses is that it is image computable , meaning that its representations can be calculated for any input image ., This makes it possible to test predictions about the internal computations of the model by generating new stimuli and running in silico experiments ., In the next two sections , we run a series of experiments on the CNN to gain insights into the factors that underlie its predictive accuracy in explaining the representations of the navigational-affordance model and the OPA ., A fundamental issue for understanding any model of sensory computation is determining the aspects of the sensory stimulus on which it operates ., In other words , what sensory inputs drive the responses of the model ?, To answer this question , we investigated the image features that drive affordance-related responses in the CNN ., Specifically , we sought to identify classes of low-level stimulus features that are critical for explaining the representational similarity of the CNN to the navigational-affordance model and the OPA ., We expected that navigational affordances would rely on image features that convey information about the spatial structure of scenes ., Our specific hypotheses were that affordance-related representations would be relatively unaffected by color information and would rely heavily on high spatial frequencies and edges at cardinal orientations ( i . e . , horizontal and vertical ) ., The hypothesis that color information would be unimportant was motivated by our intuition that color is not typically a defining feature of the structural properties of scenes and by a previous finding of ours showing that affordance representations in the OPA are partially tolerant to variations in scene textures and colors 19 ., The other two hypotheses were motivated by previous work suggesting that high spatial frequencies and cardinal orientations are especially informative for the perceptual analysis of spatial scenes , and that the PPA and possibly other scene-selective regions are particularly sensitive to these low-level visual features 33–38 , but see 39 ., To test these hypotheses , we generated new sets of filtered stimuli in which specific visual features were isolated or removed ( i . e . , color , spatial frequencies , cardinal or oblique edges; Fig 4A and 4B ) ., These filtered stimuli were passed through the CNN , and new RDMs were created for each layer ., We used the commonality-analysis technique described in the previous section to quantify the portion of the original explained variance of the CNN that could be accounted for by the filtered stimuli ., This procedure was applied to the explained variance of the CNN for predicting both the navigational-affordance RDM and the OPA RDM ( Fig 4A ) ., The results for both sets of analyses showed that over half of the explained variance of the CNN could be accounted for when the inputs contained only grayscale information , high-spatial frequencies , or edges at cardinal orientations ., In contrast , when input images containing only low-spatial frequencies or oblique edges were used , a much smaller portion of the explained variance was accounted for ., The differences in explained variance across high and low spatial frequencies and across cardinal and oblique orientations were more pronounced for the RSA predictions of the affordance RDM , but a similar pattern was observed for the OPA RDM ., We used a bootstrap resampling procedure to statistically assess these comparisons ., Specifically , we calculated bootstrap 95% confidence intervals for the following contrasts of shared-variance scores:, 1 ) high spatial frequencies minus low spatial frequencies and, 2 ) cardinal orientations minus oblique orientations ., These analyses showed that the differences in shared variance for high vs . low spatial frequencies and for cardinal vs . oblique orientations were reliable for both the affordance RDM and the OPA RDM ( all p<0 . 05 , bootstrap ) ., Together , these results suggest that visual inputs at high-spatial frequencies and cardinal orientations are important for computing the affordance-related features of the CNN ., Furthermore , these computational operations appear to be largely tolerant to the removal of color information ., Indeed , it is striking how much explained variance these inputs account for given how much information has been discarded from their corresponding filtered stimulus sets ., In addition to examining classes of input features to the CNN , we also sought to understand how inputs from different spatial positions in the image affected the similarity between the CNN and RDMs for the navigational-affordance model and the OPA ., Our hypothesis was that these RSA effects would be driven most strongly by inputs from the lower visual field ( we use the term “visual field” here because the fMRI subjects were asked to maintain central fixation throughout the experiment ) ., This was motivated by previous findings showing that the OPA has a retinotopic bias for the lower visual field 40 , 41 and the intuitive prediction that the navigational affordances of local space rely heavily on features close to the ground plane ., To test this hypothesis , we generated sets of occluded stimuli in which everything except a small horizontal slice of the image was masked ( Fig 5 ) ., These occluded stimuli were passed through the CNN , and new RDMs were created for each layer ., Once again , we used the commonality-analysis technique described above to quantify the portion of the original explained variance of the CNN that could still be accounted for by these occluded stimuli ., This procedure was repeated with the un-occluded region slightly shifted on each iteration until the entire vertical extent of the image was sampled ., We used this procedure to analyze the explained variance of the CNN for predicting both the navigational-affordance RDM and the OPA RDM ( Fig 5 ) ., For comparison , we also applied this procedure to RDMs for the other ROIs ., These analyses showed that the predictive accuracy of the CNN for both the affordance model and the OPA was driven most strongly by inputs from the lower visual field ., Strikingly , as much as 70% of the explained variance of the CNN in the OPA could be accounted for by a small horizontal band of features at the bottom of the image ( Fig 5 ) ., We created a summary statistic for this visual-field bias by calculating the difference in mean shared variance across the lower and upper halves of the image ., A comparison of this summary statistic across all tested RDMs shows that the lower visual field bias was observed for the RSA predictions of the affordance model and the OPA , but not for the other ROIs ( Fig 5 ) ., Together , these results demonstrate that information from the lower visual field is critical to the performance of the CNN in predicting the affordance RDM and the OPA RDM ., These findings are consistent with previous neuroimaging work on the retinotopic biases of the OPA 40 , 41 , and they suggest that the cortical computation of affordance-related features reflects a strong bias for inputs from the lower visual field ., The analyses above examined the stimulus inputs that drive affordance-related computations in the CNN ., We next set out to characterize the high-level features that result from these computations ., Specifically , we sought to characterize the internal representations of the CNN that best account for the representations of the OPA and the navigational-affordance model ., To do this , we performed a set of visualization analyses to reify the complex visual motifs detected by the internal units of the CNN ., We characterized the feature selectivity of CNN units using a receptive-field mapping procedure ( Fig 6A ) 42 ., The goal was to identify natural image features that drive the internal representations of the CNN ., In this procedure , the selectivity of individual CNN units was mapped across each image by iteratively occluding the inputs to the CNN ., First , the original , un-occluded image was passed through the CNN ., Then a small portion of the image was occluded with a patch of random pixel values ( 11 pixels by 11 pixels ) ., The occluded image was passed though the CNN , and the discrepancies in unit activations relative to the original image were logged ., These discrepancy values were calculated for each unit by taking the difference in magnitude between the activation to the original image and the activation to the occluded image ., After iteratively applying this procedure across all spatial positions in the image , a two-dimensional discrepancy map was generated for each unit and each image ( Fig 6A ) ., Each discrepancy map indicates the sensitivity of a CNN unit to the visual information across all spatial positions of an image ., The spatial distribution of the discrepancy effects reflects the position and extent of a unit’s receptive field , and the magnitude of the discrepancy effects reflects the sensitivity of a unit to the underlying image features ., We focused our analyses on the units in layer 5 , which was the layer with the highest RSA correlation for the both the navigational-affordance model and the OPA ., We selected 50 units in this layer based on their unit-wise RSA correlations to the navigational-affordance model and the OPA ., These units were highly informative for our effects of interest: an RDM created from just these 50 units showed comparable RSA correlations to those observed when using all units in layer 5 ( correlation with affordance RDM: r = 0 . 28; correlation with OPA RDM: r = 0 . 35 ) ., We generated receptive-field visualizations for each of these units ., These visualizations were created by identifying the top 3 images that generated the largest discrepancy values in the receptive-field mapping procedure ( i . e . , images that were strongly representative of a unit’s preferences ) ., A segmentation mask was then applied to each image by thresholding the unit’s discrepancy map at 10% of the peak discrepancy value ., Segmentations highlight the portion of the image that the unit was sensitive to ., Each segmentation is outlined in red , and regions of the image outside of the segmentation are darkened ( Fig 6B ) ., We sought to identify prominent trends across this set of receptive-field segmentations ., In a simple visual inspection of the segmentations , we detected visual motifs that were common among the units , and the results of an automated clustering procedure highlighted these trends ., Using data-driven techniques , we embedded the segmentations into a low-dimensional space and then partitioned them into clusters with similar visual motifs ., We used t-distributed stochastic neighbor embedding ( t-SNE ) to generate a two-dimensional embedding of the units based on the visual similarity of their receptive-field segmentations ( Fig 6B ) ., We then used k-means clustering to identify sets of units with similar embeddings ., The number of clusters was set at 7 based on the outcome of a cluster-evaluation procedure ., The specific cluster assignments do not necessarily indicate major qualitative distinctions between units ., Rather , they provide a data-driven means of reducing the complexity of the results and highlighting the broad themes in the data ., These themes can also be seen in the complete set of visualizations plotted in S1–S7 Figs ., These visualizations revealed two broad visual motifs: boundary-defining junctions and large , extended surfaces ., Boundary-defining junctions are the regions of an image where two or more extended planes meet ( e . g . , clusters 1 , 5 , 6 , and 7 in Fig 6B ) ., These were often the junctions of walls and floors , and less often ceilings ., This was the most common visual motif across all segmentations ., Large , extended surfaces were uninterrupted portions of floor and wall planes ( e . g . , cluster 3 in Fig 6B ) ., There were also units that detected more complex structural features that were often indicative of doorways and other open pathways ( e . g . , clusters 2 and 4 in Fig 6B ) ., A common thread running through all these visualizations is that they appear to reflect high-level scene features that could be reliably used to map out the spatial layout and navigational affordances of the local environment ., Boundary-defining junctions and large , extended surfaces provide critical information about the spatial geometry of the local scene , and more fine-grained structural elements , such as doorways and open pathways , are critical to the navigational layout of a scene ., Together , these results suggest a minimal set of high-level visual features that are critical for modeling the navigational affordances of natural images and predicting the affordance-related responses of scene-selective visual cortex ., Our analyses thus far have focused on a carefully selected set of indoor scenes in which the potential for navigation was clearly delimited by the spatial layout of impassable boundaries and solid open ground ., Indeed , the built environments depicted in our stimuli were designed so that humans could readily navigate through them ., However , there are many environments in which navigability is determined by a more complex set of perceptual factors ., For example , in outdoor scenes navigability can be strongly influenced by the material properties of the ground plane ( e . g . , grass , water ) ., We wondered whether the components of the CNN that were related to the navigational-affordance properties of our indoor scenes could be used to identify navigational properties in a broader range of images ., To address this question , we examined a set of images depicting natural landscapes , whose navigational properties had been quantified in a previous behavioral study 43 ., Specifically , these stimuli included 100 images that could be grouped into categories of low or high navigability based on subjective behavioral assessments ., The overall navigability of the images reflected subjects’ judgments of how easily they could move through the scene ., These navigability assessments were influenced by a wide range of scene features , including the spatial layout of pathways and boundaries , the presence of clutter and obstacles , and the potential for treacherous conditions ., There were also low and high categories for 13 other scene properties ( Fig 7A ) ., Each scene property was associated with 100 images ( 50 low and 50 high ) , and many images were used for multiple scene properties ( 548 images in total ) ., We sought to determine whether the units from the CNN that were highly informative for identifying the navigational affordances of indoor scenes could also discern the navigational properties in this heterogeneous set of natural landscapes ., To do this , we focused on the 50 units selected for the visualization analyses in Fig 6 , and we used the responses of these units to classify natural landscapes based on their overall navigability ( Fig 7A ) ., We found that not only were these CNN units able to classify navigability across a broad range of outdoor scenes , but they also appeared to be particularly informative for this task relative to the
Biologically inspired deep convolutional neural networks ( CNNs ) , trained for computer vision tasks , have been found to predict cortical responses with remarkable accuracy ., However , the internal operations of these models remain poorly understood , and the factors that account for their success are unknown ., Here we develop a set of techniques for using CNNs to gain insights into the computational mechanisms underlying cortical responses ., We focused on responses in the occipital place area ( OPA ) , a scene-selective region of dorsal occipitoparietal cortex ., In a previous study , we showed that fMRI activation patterns in the OPA contain information about the navigational affordances of scenes; that is , information about where one can and cannot move within the immediate environment ., We hypothesized that this affordance information could be extracted using a set of purely feedforward computations ., To test this idea , we examined a deep CNN with a feedforward architecture that had been previously trained for scene classification ., We found that responses in the CNN to scene images were highly predictive of fMRI responses in the OPA ., Moreover the CNN accounted for the portion of OPA variance relating to the navigational affordances of scenes ., The CNN could thus serve as an image-computable candidate model of affordance-related responses in the OPA ., We then ran a series of in silico experiments on this model to gain insights into its internal operations ., These analyses showed that the computation of affordance-related features relied heavily on visual information at high-spatial frequencies and cardinal orientations , both of which have previously been identified as low-level stimulus preferences of scene-selective visual cortex ., These computations also exhibited a strong preference for information in the lower visual field , which is consistent with known retinotopic biases in the OPA ., Visualizations of feature selectivity within the CNN suggested that affordance-based responses encoded features that define the layout of the spatial environment , such as boundary-defining junctions and large extended surfaces ., Together , these results map the sensory functions of the OPA onto a fully quantitative model that provides insights into its visual computations ., More broadly , they advance integrative techniques for understanding visual cortex across multiple level of analysis: from the identification of cortical sensory functions to the modeling of their underlying algorithms .
How does visual cortex compute behaviorally relevant properties of the local environment from sensory inputs ?, For decades , computational models have been able to explain only the earliest stages of biological vision , but recent advances in deep neural networks have yielded a breakthrough in the modeling of high-level visual cortex ., However , these models are not explicitly designed for testing neurobiological theories , and , like the brain itself , their internal operations remain poorly understood ., We examined a deep neural network for insights into the cortical representation of navigational affordances in visual scenes ., In doing so , we developed a set of high-throughput techniques and statistical tools that are broadly useful for relating the internal operations of neural networks with the information processes of the brain ., Our findings demonstrate that a deep neural network with purely feedforward computations can account for the processing of navigational layout in high-level visual cortex ., We next performed a series of experiments and visualization analyses on this neural network ., These analyses characterized a set of stimulus input features that may be critical for computing navigationally related cortical representations , and they identified a set of high-level , complex scene features that may serve as a basis set for the cortical coding of navigational layout ., These findings suggest a computational mechanism through which high-level visual cortex might encode the spatial structure of the local navigational environment , and they demonstrate an experimental approach for leveraging the power of deep neural networks to understand the visual computations of the brain .
medicine and health sciences, diagnostic radiology, functional magnetic resonance imaging, neural networks, engineering and technology, applied mathematics, brain, social sciences, neuroscience, magnetic resonance imaging, algorithms, simulation and modeling, mathematics, brain mapping, computational neuroscience, vision, neuroimaging, coding mechanisms, research and analysis methods, computer and information sciences, imaging techniques, visual cortex, navigation, psychology, radiology and imaging, diagnostic medicine, anatomy, biology and life sciences, sensory perception, physical sciences, computational biology