Representations of regular and irregular shapes by deep Convolutional Neural Networks, monkey inferotemporal neurons and human judgments
Recently , several studies compared the representations of visual images in deep Convolutional Neural Networks ( CNN ) with those of biological systems , such as the primate ventral visual stream 1–4 ., These studies showed that the representation of visual objects in macaque inferior temporal ( IT ) cortex corresponds better with the representations of these images in deep CNN layers than with representations of older computational models such as HMAX 5 ., Similar findings were obtained with human fMRI data 6–10 ., The images used in these studies were those of real objects in cluttered scenes , which are the same class of images as those employed to train the deep CNNs for classification ., Other single unit studies of IT neurons employed two-dimensional ( 2D ) shapes and observed highly selective responses to such stimuli ( for review see 11 ) ., If deep CNNs provide a realistic model of IT responses , then the CNNs should capture also the selectivity observed for such two-dimensional shapes in IT ., To our knowledge , thus far there has been no comparison between the 2D-shape representation of IT neurons , measured with such reduced stimuli , and that of deep CNN models ., It is impossible to predict from existing studies that compared deep CNN activations and neurophysiology whether the deep CNNs , which are trained with natural images , can faithfully model the selectivity of IT neurons for two-dimensional abstract shapes ., Nonetheless , such correspondence between CNN models and single unit selectivity for abstract shapes is critical for assessing the generalizability of CNN models to stimuli that differ markedly from those of the trained task but have been shown to drive selectively IT neurons ., Previously , we showed that a linear combination of units of deep convolutional layers of CNNs trained with natural images could predict reasonably well the shape selectivity of single neurons recorded from an fMRI-defined body patch 4 ., However , in that study , we adapted for each single unit the shapes to the shape preference of that neuron , precluding a comparison between the shape representation of the population of IT neurons and deep CNNs ., To perform such a comparison , one should measure the responses of IT neurons to the same set of shapes ., Furthermore , the shape set should include variations in shape properties IT neurons were shown to be sensitive to ., Also , the IT response selectivities for such shapes should not trivially be explainable by physical image similarities , such as pixel-based differences in graylevels ., Kayaert et al . 12 measured the responses of single IT neurons to a set of shapes that varied in regularity and the presence of curved versus straight boundaries ( Fig 1 ) ., The first group of stimuli of 12 was composed of regular geometric shapes ( shown in the first two rows of Fig 1 and denoted as Regular ( R ) ) that all have at least one axis of symmetry ., These shapes are simple , i . e . , have low medial axis complexity 13 ., The stimulus pairs in each column of these two rows ( denoted by a and b ) differed in a non-accidental property ( NAP ) ., NAPs are stimulus properties that are relatively invariant with orientation in depth , such as whether a contour is straight or curved or whether a pair of edges is parallel or not ., These properties can allow efficient object recognition at different orientations in depth not previously experienced 14–16 ., NAPs can be contrasted with metric properties ( MPs ) , which vary with orientation in depth , such as aspect ratio or the degree of curvature ., The three other groups are all ‘Irregular’ ., They differed from the Regular shapes in that they do not have a single axis of symmetry ., The two shapes in each row of the three Irregular groups differed in the configuration of their concavities and convexities or corners ., The shapes in the Irregular Simple Curved ( ISC ) set all had curved contours ., The Irregular Simple Straight ( ISS ) shapes were derived from the ISC shapes by replacing the curved contours with straight lines ., Thus , the corresponding stimuli in the ISS and ISC shapes differed in a NAP ., Last , the Irregular Complex ( IC ) group was more complex in that the shapes in that group had a greater number of contours ., Kayaert et al . 12 found that anterior IT neurons distinguished the four groups of shapes ., Importantly , the differences in IT responses amongst the shapes could not be explained by pixel-based gray level differences , nor by HMAX C2 unit differences ., In fact , none of the tested quantitative models of object processing could explain the IT response modulations ., Furthermore , the IT response modulations were greater for the Regular shapes and when comparing the curved and straight Irregular Simple shapes than within the 3 Irregular shape groups , suggesting a greater sensitivity for NAPs than for MPs ( see also 17 , 18 ) ., We reasoned that this shape set and corresponding IT responses was useful to examine to what degree different layers of deep CNNs and IT neurons represent abstract shapes similarly ., We employed deep CNNs that were pretrained to classify ImageNet data 19 , consisting of images of natural objects in scenes ., Hence , the CNNs were not exposed during training to silhouette shapes shown to the IT neurons ., Deep CNNs have a particular architecture with early units having small receptive fields , nonlinear pooling of units of the previous layer , etc ., Such a serial , hierarchical network architecture with increasing receptive field size across layers may result in itself , i . e . without training , in changes in the representational similarity across layers ., To assess whether potential correlations between IT and CNN layer response modulations resulted from classification training or from the CNN architecture per se , we also compared the activations of untrained CNNs with the IT response modulations ., Kayaert et al . 12 had also human subjects sort the same shapes based on similarity and found that human subjects had a pronounced higher sensitivity to the difference between the curved and straight simple irregular shapes ( relative to the regular shapes ) than the IT neurons ., We examined whether a similar difference in response pattern between macaque IT neurons and human similarity judgements would emerge in the deep CNNs ., We expected that deeper layers would resemble the human response patterns while the IT response pattern would peak at less deep layers ., Kayaert et al 12 recorded the responses of 119 IT neurons to the 64 shapes shown in Fig 1 ., The 64 shapes are divided in four groups based on their regularity , complexity and whether they differed in NAPs ., We presented the same shapes to 3 deep CNNs: Alexnet 20 , VGG-16 , VGG-19 21 and measured the activations of the units in each layer of the deep nets ., These deep nets differ in their number of layers , the number of units in each layer and the presence of a normalization stage , but each have rectifying non-linearity ( RELU ) and max pooling stages ( Fig 2 ) ., We employed deep nets that were pre-trained in classification of a database of natural images , which were very different in nature from the abstract shape stimuli that we employ here to test the models and neurons ., The aim was to compare the representations of the shapes between IT neurons and each layer of the deep nets ., To do this , we employed representational similarity analyses 22 , 23 , following the logic of second order isomorphism 24 , 25 , and examined the correlation between the neural IT-based similarities and CNN-based similarities in responses to shapes ., We are not trying to reconstruct the shapes based on IT neuron or CNN unit outputs but we are examining whether shapes that are represented close to each other in the neural IT space are also represented close to each other in the CNN layer space ., In a first analysis , we computed the pairwise dissimilarity between all 64 stimuli using the responses of the IT neurons and the activations in each of the CNN layers ., We employed two dissimilarity metrics: Euclidean distance and 1 –Spearman rank correlation ρ ., The dissimilarity matrices computed with the Euclidean distance metric for the IT neurons and for 5 layers of the trained CNNs are illustrated in Fig 3B and 3C , respectively ., In this and the next figures , we will show only the data for Alexnet and VGG-19 , since VGG-16 and VGG19 produced similar results ., In addition , Fig 3A shows the pixel-based dissimilarities for all image pairs ., Visual inspection of the dissimilarity matrices suggests that ( 1 ) the pattern of dissimilarities changes from the superficial to deep layers in a relatively similar way in the CNNs , ( 2 ) the dissimilarity matrix of the first layer ( e . g . conv1 . 1 ) resembles the pixel-based similarities ( Fig 3A ) and ( 3 ) the deeper layers resemble more the IT neural data ( Fig 3B ) ., We quantified the similarity between the IT shape representation and that of each layer by computing the Spearman Rank correlation between the corresponding pairwise dissimilarities of IT and each layer ., Thus , we could assess to what degree stimuli that produce a very different ( similar ) pattern of responses in IT also show a different ( similar ) pattern of activations in a CNN layer ., We found that for both dissimilarity metrics the similarity between IT neuronal responses and trained CNN layer activations increased significantly with the depth of the layer ., This is shown using the Euclidean distance metric for Alexnet and VGG-19 in Fig 4 ( see S1 Fig for the data of both distance metrics and the 3 networks ) ., In the VGG nets , the similarity peaked at the deepest convolutional layers ( Fig, 4 ) and then decreased for the deepest layers ., In fact , the Spearman correlations for the last two fully connected layers did not differ significantly from that of the first convolutional layer in each CNN ( Fig 4 ) ., The decrease in similarity for the deepest layers was weaker in Alexnet ., The peak similarity was similar amongst the 3 nets , with ρ hovering around 0 . 60 , and were larger for the correlation ( mean peak ρ = 0 . 64 ) compared with Euclidean distance metric ( mean peak ρ = 0 . 58 ) ., To assess the degree to which the models explained the neural data , we computed the reliability of the neural-based distances giving the finite sampling of the IT neuron population ., This noise ceiling was computed by randomly splitting the neurons into two groups , computing the dissimilarities for each group , followed by computation of the Spearman rank correlation between the dissimilarities of the two groups ., This split-half reliability computation was performed for 10000 random splits ., Fig 4 shows the 2 . 5 , 50 ( median ) and 97 . 5 percentiles of the Spearman-Brown corrected correlations between the two groups ., The correlations between ( some ) CNN layers and neural responses were close but still below the estimated noise of the neural dissimilarities ., In order to assess to what degree the similarity between neural data and the CNN layers reflects the architecture of the CNNs versus image classification training , we computed also the similarity for untrained networks with random weights ., Fig 3C illustrates dissimilarity matrices computed using Euclidean distances for 5 untrained layers of two CNNs ., Visual inspection suggests little change in the dissimilarity matrices of the different layers of the CNNs , except for fc8 ., Furthermore , the pattern of dissimilarities resembled the pixel-based dissimilarities shown in Fig 3A ., Both observations were confirmed by the quantitative analysis ., The Spearman correlations of the neural data and untrained CNNs increased only weakly with depth , except for a marked decrease in correlation for the last two fully connected layers ., Except for the deep convolutional and the last two layers , the trained and untrained networks showed similar Spearman correlations of the neural and CNN distances ( Fig 4 ) ., This suggests that overall the similarity between the IT data and the shallow CNN layers are unrelated to classification training but reflect merely the CNN architecture ., Significant differences between trained and untrained CNNs were observed for the deeper convolutional layers ( Fig 4 ) , suggesting that the similarity between IT and the deep convolutional layers depends on classification training ., The similarities for the first fully connected layer ( fc6 and relu6 in Fig, 4 ) did not differ significantly between the trained and untrained layers ( except for the correlation metric in AlexNet ( S1 Fig ) ., The deepest two ( fully connected ) layers showed again a significantly greater similarity for the trained compared with the untrained networks ., However , this can be the result of the sharp drop in correlations for these layers in the untrained network ., Overall , these data suggest that the shape representations of the trained deep convolutional layers , but not of the deepest layers , shows the highest similarity with shape representations in macaque IT ., Receptive field ( RF ) size increases along the layers of the CNNs , allowing deeper layer units to integrate information from larger spatial regions ., The difference in IT-CNN similarity between untrained and trained layers shows that the increase in RF size cannot by itself explain the increased IT-CNN similarity in deeper layers , since untrained CNN also increase their RFs along the layer hierarchy ., Also , the decrease in similarity between IT responses and the fully connected layers argues against RF size being the mere factor ., Nonetheless , although not the only contributing factor , RF size is expected to matter since arguably small RFs cannot capture overall shape when the shape is relatively large ., Hence , it is possible that the degree of IT-CNN similarity for different layers depends on shape size , with smaller shapes showing a greater IT-CNN similarity at earlier layers ., We tested this by computing the activations to shapes that were reduced in size by a factor of two in all layers of each of the 3 trained CNNs ., Fig 5 compares the correlations between dissimilarities of the trained Alexnet and VGG-19 networks and IT dissimilarities for the original and reduced sizes , with dissimilarities computed using Euclidean distances ., The stars indicate significant differences between the similarities for the two sizes ( tested with a FDR corrected randomization test; same procedure as in Fig 4 when comparing trained and untrained correlations ) ., In each of the CNNs ( S2 Fig ) , the IT-CNN similarity increased at more superficial layers for the smaller shape ., The overall peak IT-CNN similarity was highly similar for the two sizes in the VGG networks and occurred at the deep convolutional layers ., For Alexnet , the overall similarity was significantly higher for the smaller shapes in the deep layers ., This analysis indicates that shape size is a contributing factor that determines at which layer the IT-CNN similarity increases , but that for the VGG networks , peak similarity in the deep layers does not depend on size ( at least not for the twofold variation in size employed here ) ., Note that also for the smaller size the IT-CNN similarity drops markedly for the fully connected layers in the VGG networks ., Thus , the overall trends are independent of a twofold change in shape size ., In the preceding analyses , we included all units of each CNN layer ., To examine whether the similarity between the CNN layers and the IT responses depends on a relatively small number of CNN units or is distributed amongst many units , we reran the representational similarity analysis of deep CNN layers and IT neurons for the whole shape set for smaller samples of CNN units ., We took for each network the layer showing the peak IT-CNN similarity and for that layer sampled 10000 times at random a fixed percentage of units ., We restricted the population of units to those that showed a differential activation ( standard deviation of activation across stimuli greater than 0 ) since only those can contribute to the Euclidean distance ., Fig 6A plots the median and 95% range of Spearman rank correlation coefficients between IT and CNN layer dissimilarities for the whole shape set as a function of the percent of sampled units for two CNNs ., We found that the IT-CNN similarity was quite robust to the number of sampled units ., For instance , for Alexnet , the IT-CNN similarity for the original and the 95% range of the 10% samples overlap , indicating that 315 Alexnet units can produce the same IT-CNN similarity as the full population of units ., Note also that the lower bound of the 95% range is still above the IT-CNN similarities observed for the untrained network ( median Spearman rho about 0 . 40; see Fig 4 ) ., This indicates that the IT-CNN similarity does not depend on a small subset of units , since otherwise the range of similarities ( Spearman rho correlations ) for the 10% samples would be much greater ., The same holds for the other CNNs ( S3 Fig ) , except that these tolerated even smaller percent sample size ( for VGG19 even 0 . 1% , which corresponds to 100 units ) ., The above analysis appears to suggest that the activations of the CNN units to the shapes are highly correlated with each other ., To address this directly , we performed Principal Component Analysis ( PCA ) of unit activations of the same peak CNN layers as in Fig 6 and computed Euclidean distance based dissimilarities between all stimulus pairs for the first , first two , etc . principal components ( PCs ) , followed by correlation with the neural dissimilarities as done before for the distances computed across all units of a CNN layer ., For both the Alexnet and VGG-19 layer , the first 10 PCs explained about 70% of the variance in CNN unit activations to the 64 stimuli ( Fig 7B ) ., Only the first 3 ( Alexnet ) or 5 ( VGG-19 ) PCs were required to obtain a similar correlation between the model and neural distances as observed when using all model units of the layer ( Fig 7A; about 7 PCs were required for VGG-16; see S4 Fig ) ., This analysis shows that the neural distances between the abstract shapes relate to a relatively low dimensional shape representation in the CNN layer , with a high redundancy between the CNN units ., In the above analyses , we compared the overall similarity of the shape representations in IT and CNN layers ., However , a more stringent comparison between the shape representations in IT and the CNNs involves response modulations for the shape pairs for which Kayaert et al 12 observed striking differences between predictions of pixel-based models or computational models like HMAX and the neural responses ., The average response modulations ( quantified by pairwise Euclidean distances ) for the different group pairs comparisons are shown in Fig 8 for the IT neural data , the HMAX C2 layer and the pixel differences ., Kayaert et al 12 showed that the mean response modulation in IT ( Fig 8A ) was significantly greater for the regular shape pairs ( 1–8 in Fig, 1 ) than for the 3 irregular shape group pairs , despite the pixel differences between members of a pair being , on average , lower or similar for the regular group than for the 3 irregular groups ( Fig 8D ) ., In addition , the response modulation to ISC vs . ISS was significantly greater than the modulations within IC , ISC and ISS , although the average pixel-difference within the ISC vs . ISS-pairs was much lower than the pixel-differences within the other pairs ., This differential neural response modulation to ISC vs ISS was present for both members of the ISC and ISS pairs ( a and b members: “ISCa vs ISSa” and “ISCb vs ISSb” ) and thus was highly reliable ., Note that the difference between ISC vs . ISS and the IC and ISS shape groups that are present in the neural data is not present for the HMAX C2 distances ( Fig 8C ) ., Kayaert et al . 12 reported also a relatively higher sensitivity to the straight vs . curved contrast of the ISC vs . ISS comparison compared with the regular shapes in human similarity ratings ( Fig 8B ) , compared with the IT neural data ., In other words , human subjects appear to be more sensitive to the curved versus straight NAP difference than macaque IT neurons ., In a second analysis , we determined whether the marked differences in IT response modulations and human judgements shown in Fig 8 are present in the dissimilarities for the different layers of the deep CNNs ., Fig 9 illustrates the results for 8 layers of VGG-19 ., The left column of the figure plots the distances for the trained network ., The dissimilarities for the first convolutional layer fits the pixel-based distances amongst the shape pairs ( Fig 8D; Pearson correlation between pixel-based distances and first layer distances = 0 . 966 ) , but differ from those observed in IT and for human judgements ., Similar trends are present until the very deep convolutional layers where the dissimilarities became strikingly similar to those observed in macaque IT ( e . g . compare trained conv5 . 4 or pool5 of Fig 9 with Fig 8A ) ., The dissimilarities for the last two layers ( e . g . trained relu7 and fc8 in Fig, 9 ) are strikingly similar to those observed for the human judgements ( Fig 8B ) , and differ from the pattern seen in macaque IT neurons ., Indeed , as noted above , the human judgements differ from the IT responses in their sensitivity for the ISC vs ISS comparison relative to that for the regular shape pairs: for the human judgement distances , the ISC vs ISS distances are greater than for the regular shape distances while for the neural distances both are statistically indistinguishable ( Kayaert et al . 12 ) ., Therefore , we tested statistically for which CNN layer the ISC vs ISS distances were significantly greater than the regular shape distances ( Wilcoxon test ) , thus mimicking the human distances ., We found a significant difference for the very deep VGG19 layer fc8 ( p = 0 . 039 ) and VGG16 layers fc7 ( p = 0 . 039 ) , relu7 ( p = 0 . 023 ) , and fc8 ( p = 0 . 023 ) ., Although the deepest Alexnet ( fully connected ) layers showed the same trend , this failed to reach significance ., These tests showed that only the very deep CNN layers mimicked the human judgements ., None of the untrained CNN layers showed a dissimilarity profile similar to that observed in monkey IT or in human judgements ( Fig 9 , right column ) ., In fact , the untrained data resembled more the pixel-based distances ( see Fig 8D ) ., Indeed , the Pearson correlation between the pixel-based distances and the conv1 . 1 distances was 0 . 999 for the untrained VGG-19 ., We quantified the correspondence between the neural response dissimilarities of Fig 8A and the CNN layer dissimilarities ( as in Fig, 9 ) by computing the Pearson correlation coefficient between the dissimilarity profiles ( n = 6 dissimilarity pairs ) ., The same quantification was performed for the human judgements ( Fig 8B ) and the CNN dissimilarities ( n = 5 pairs ) ., These correlations are plotted in Fig 10A and 10B as a function of layer for two CNNs , trained and untrained ., For the neural data , the correlations are negative for the shallow layers and highly similar for the trained and untrained CNNs ., The negative correlations are a result of the nearly inverse relationship between neural and low-level ( pixel ) differences between the shapes ( Fig 8D ) ., This was not accidental , but by design: when creating the stimuli , Kayaert et al 12 ensured that the NAP differences ( e . g . between ISC and ISS ) were smaller than MP differences ., For both VGG networks ( S5 Fig; Fig 10B ) , there was a sharp increase in correlations at the trained deep 5 . 1 convolutional layer , followed by a decrease of the correlations for the fully connected layers ., This trend was similar , although more abrupt , to that observed for the global dissimilarities of Fig 4 ., For Alexnet , the increase of the correlations with increasing depth of the trained convolutional layers was more gradual , but like the VGG networks , high correlations were observed for the deeper trained convolutional layers ., For the human judgement data , the correlations were already higher for the trained compared with the untrained CNNs at the shallow layers , although still negative ., Like the neural data , there was a marked increase in correlation at the very deep trained layers ., Contrary to the neural data , the correlations for the human judgements continued to increase along the trained fully connected layers , approaching a correlation of 1 at the last layer ., These data show that the average response modulations of IT neurons for the shape groups of Fig 1 correspond nearly perfectly with those of the deeper layers of CNNs , while the differences in human similarity judgements between the groups are captured by the later fully connected layers of the CNNs ., This holds for Alexnet and VGG nets ., Note that the deep CNN layers performed better at predicting the neural IT and human perceptual dissimilarities than the HMAX C2 layer output ( Fig 10C ) ., As for the representational similarity analysis for all shapes ( Fig 6A ) , we computed also the Pearson correlation coefficients between the dissimilarity profiles ( n = 6 dissimilarity pairs ) of the same peak CNN layers and the IT distances for the 6 shape groups ( as in Fig, 10 ) for smaller samples of units ., As shown in Fig 6B , we observed similar IT-CNN correlations for the within-group distances up to the 1% and 0 . 1% samples compared with the full population of units for Alexnet and VGG , respectively ., Again , this suggests that IT-CNN similarity does not depend on a small number of outlier CNN units ., The greater tolerance for percent sample size for the VGG units is because the VGG layers consisted of a larger number of units per se ( total number of units are indicated in the legend of Fig 6 ) ., In addition , we computed the mean distances for the same layers and their correlation with the mean neural modulations as a function of retained PCs ( Fig 7B ) ., Up to 30 PCs were required to obtain a similar correlation between neural and CNN layer distances for the six groups of shapes as when including all units of the layer ( Fig 7B ) ., This suggests that the close to perfect modeling of the mean response modulations across the 6 shape groups required a relatively high dimensional representation of the shapes within the CNN layer ., The particular set of shapes that we employed in the present study was designed originally to test the idea that the shape selectivity of IT neurons reflects the computational challenges posed when differentiating objects at different orientations in depth 12 , 14 ., Here , we show that deep CNNs that were trained to classify a large set of natural images show response modulations to these shapes that are similar to those observed in macaque IT neurons ., We show that untrained CNNs with the same architecture than the trained CNNs , but with random weights , demonstrate a poorer IT-CNN similarity than the CNNs trained in classification ., The difference between the trained and untrained CNNs emerged at the deep convolutional layers , where the similarity between the shape-related response modulations of IT neurons and the trained CNNs was high ., Unlike macaque IT neurons , human similarity judgements of the same shapes correlated best with the deepest layers of the trained CNNs ., Early and many later studies of IT neurons employed shapes as stimuli ( e . g . 26–31 , 22 , 32–37 ) , in keeping with shape being an essential object property for identification and categorization ., Deep CNNs are trained with natural images of objects in cluttered scenes ., If deep CNNs are useful models of biological object recognition 38 , their shape representations should mimic those of the biological system , although the CNNs were not trained with such isolated shapes ., We show here that indeed the representation of the response modulations by rather abstract , unnatural shapes is highly similar for deep CNN layers and macaque IT neurons ., Note that the parameters of these CNN models are set via supervised machine learning methods to do a task ( i . e . classify objects ) rather than to replicate the properties of the neural responses , as done in classic computational modeling of neural selectivities ., Thus , the same CNN model that fits neural responses to natural images 1–4 also simulates the selectivity of IT neurons for abstract shapes , demonstrating that these models show generalization across highly different stimulus families ., Of course , the high similarity between deep CNN layers and IT neurons activation patterns we show here may not generalize for ( perhaps less fundamental ) shape properties that we did not vary in our study ., Kubilius et al . 39 showed that deep nets captured shape sensitivities of human observers ., They showed that deep Nets , in particular their deeper layers , show a NAP advantage for objects ( “geons” ) , as does human perception ( and macaque IT 18 ) ., Although we also manipulated NAPs , our shapes differed in addition in other properties such as regularity and complexity ., Furthermore , our shapes are unlike real objects and more abstract than the shaded 3D objects employed by Kubilius et al . 39 when manipulating NAPs ., In both the representational similarity analysis and the response modulations comparisons amongst shape groups , we found that the correspondence between IT and deep CNN layers peaked at the deep convolutional layers and then decreased for the deeper layers ., Recently , we observed a similar pattern when using deep CNN activations of individual layers to model the shape selectivity of single neurons of the middle Superior Temporal Sulcus body patch 4 , a fMRI-defined region of IT that is located posterior with respect to the present recordings ., The increase with deeper layers of the fit between CNN activations and neural responses has also been observed when predicting with CNN layers macaque IT multi-unit selectivity 40 , voxel activations in human LO 9 and the representational similarity of macaque and human ( putative ) IT 8 , 10 using natural images ., However , the decrease in correlation between CNNs and neural data that we observed for the deepest layers was not found in fMRI studies that examined human putative IT 8 , 10 , although such a trend was present in 6 when predicting CNN features from fMRI activations ., The deepest layers are close to or at the categorization stage of the CNN and hence strongly dependent on the classifications the network was trained on ., The relatively poor performance of the last layers is in line with previous findings that IT neurons show little invariance across exemplars of the same semantic category 41 , 42 , unlike the deepest CNN units 43 ., The question of what the different layers in the various CNN models with different depths represent neurally remains basically unanswered ., Shallow CNN layers can be related to early visual areas ( e . g . V1; V4 ) and deeper layers to late areas ( e . g . IT ) ., However , different laminae within the same visual area ( e . g . input and output layers ) may also correspond to different layers of CNNs ., Furthermore , units of a single CNN layer may not correspond to a single area , but the mapping might be more mixed with some units of different CNN layers being mapped to area 1 , while other units of partially overlapping CNN layers to area 2 , etc ., Finally , different CNN layers may represent different temporal processing stages within an area , although this may map partially to the different laminae within an area ., Further research in which recordings in different laminae of several areas will be obtained for the same stimulus sets , followed by mapping these to units of different layers in various CNNs , may start to answer this complex issue ., In contrast with IT neurons , human similarity judgements of our shapes
Recent studies suggest that deep Convolutional Neural Network ( CNN ) models show higher representational similarity , compared to any other existing object recognition models , with macaque inferior temporal ( IT ) cortical responses , human ventral stream fMRI activations and human object recognition ., These studies employed natural images of objects ., A long research tradition employed abstract shapes to probe the selectivity of IT neurons ., If CNN models provide a realistic model of IT responses , then they should capture the IT selectivity for such shapes ., Here , we compare the activations of CNN units to a stimulus set of 2D regular and irregular shapes with the response selectivity of macaque IT neurons and with human similarity judgements ., The shape set consisted of regular shapes that differed in nonaccidental properties , and irregular , asymmetrical shapes with curved or straight boundaries ., We found that deep CNNs ( Alexnet , VGG-16 and VGG-19 ) that were trained to classify natural images show response modulations to these shapes that were similar to those of IT neurons ., Untrained CNNs with the same architecture than trained CNNs , but with random weights , demonstrated a poorer similarity than CNNs trained in classification ., The difference between the trained and untrained CNNs emerged at the deep convolutional layers , where the similarity between the shape-related response modulations of IT neurons and the trained CNNs was high ., Unlike IT neurons , human similarity judgements of the same shapes correlated best with the last layers of the trained CNNs ., In particular , these deepest layers showed an enhanced sensitivity for straight versus curved irregular shapes , similar to that shown in human shape judgments ., In conclusion , the representations of abstract shape similarity are highly comparable between macaque IT neurons and deep convolutional layers of CNNs that were trained to classify natural images , while human shape similarity judgments correlate better with the deepest layers .
The primate inferior temporal ( IT ) cortex is considered to be the final stage of visual processing that allows for object recognition , identification and categorization of objects ., Electrophysiology studies suggest that an object’s shape is a strong determinant of the neuronal response patterns in IT ., Here we examine whether deep Convolutional Neural Networks ( CNNs ) , that were trained to classify natural images of objects , show response modulations for abstract shapes similar to those of macaque IT neurons ., For trained and untrained versions of three state-of-the-art CNNs , we assessed the response modulations for a set of 2D shapes at each of their stages and compared these to those of a population of macaque IT neurons and human shape similarity judgements ., We show that an IT-like representation of similarity amongst 2D abstract shapes develops in the deep convolutional CNN layers when these are trained to classify natural images ., Our results reveal a high correspondence between the representation of shape similarity of deep trained CNN stages and macaque IT neurons and an analogous correspondence of the last trained CNN stages with shape similarity as judged by humans .
medicine and health sciences, diagnostic radiology, functional magnetic resonance imaging, neural networks, visual object recognition, vertebrates, social sciences, neuroscience, animals, mammals, learning and memory, magnetic resonance imaging, primates, perception, cognitive psychology, cognition, brain mapping, memory, vision, neuroimaging, old world monkeys, research and analysis methods, computer and information sciences, imaging techniques, monkeys, animal cells, macaque, cellular neuroscience, psychology, eukaryota, diagnostic medicine, radiology and imaging, cell biology, neurons, biology and life sciences, cellular types, sensory perception, cognitive science, amniotes, organisms