How Deep is the Feature Analysis underlying Rapid Visual Categorization?
Rapid categorization paradigms have a long history in experimental psychology: Characterized by short presentation times and speeded behavioral responses, these tasks highlight the efficiency with which our visual system processes natural object categories. Previous studies have shown that feed-forward hierarchical models of the visual cortex provide a good fit to human visual decisions. At the same time, recent work in computer vision has demonstrated significant gains in object recognition accuracy with increasingly deep hierarchical architectures. But it is unclear how well these models account for human visual decisions and what they may reveal about the underlying brain processes. We have conducted a large-scale psychophysics study to assess the correlation between computational models and human behavioral responses on a rapid animal vs. non-animal categorization task. We considered visual representations of varying complexity by analyzing the output of different stages of processing in three stateof-the-art deep networks. We found that recognition accuracy increases with higher stages of visual processing (higher level stages indeed outperforming human participants on the same task) but that human decisions agree best with predictions from intermediate stages.
Overall, these results suggest that human participants may rely on visual features of intermediate complexity and that the complexity of visual representations afforded by modern deep network models may exceed the complexity of those used by human participants during rapid categorization.
The paper compares performance of deep networks to psychophysical performance of humans performing classification on the same images.  From the correlation in performance it is claimed that human performance is best modeled based on intermediate levels of a deep net. 
Overall this is an interesting study, and I do appreciate the method of using Mechanical Turk to collect data. As far as I know the result is novel, that humans correlate most strongly with intermediate layers of a variety of different deep nets. I think the Discussion section is a bit to grandiose in claiming that layer 10 of their fine-tuned VGG-16 model presents some sort of supporting evidence of the Serre et al claim about how many processing stages humans have. I also would have liked to see an investigation of what the representations at 70% total depth were for the different networks. It would have supported their argument better if they could have asserted that the features at 70% depth in a smaller network, such as AlexNet corresponded strongly with the features found in a larger net, such as VGG-19.   Specific  comments:  As far as I could tell they did not use a mask for the human subjects.  Why?   If we assume that the mask serves the purpose it is intended to serve, then it should stop contributions of recurrent and top-down processing from influencing the human's response. I think using a mask would increase support for their argument that error correlations can be used to assess the "depth" of the human recognition system.  The authors compared the human results to results from three staple deep nets: AlexNet, VGG16, and VGG19. For convolutional layers they grab a random sample of features from each layer. They fix the number of features they use, which I find a little concerning. Given that deeper layers in these networks have fewer features, if they fix the sample size then they are going to get a more complete set of features as they ascend the hierarchy, and thus a more complete description of the layer's representation. Given the features from a certain Deep Net layer, they use a SVM to perform the 2AFC task. This surprises me, I would think that they would have just used a couple of fully connected layers as is done in the ImageNet challenge.   They find that the humans and deep nets have the highest response correlation when using features from layers at 70% of the model's depth. It seems like this is consistent across models, which is a bit confusing. The conclusion they draw from the result is that humans use intermediate-level features. However, it is based on the assumption that the features at whatever layer corresponds to 70% of the model's depth are uniform across models. They do not discuss this directly, and I would be very interested to see if it is true at all. Does modifying the total model depth have any influence on the features learned at a relative depth?  They further note that the maximum correlation observed with one of their models (fined tuned VGG16) occurs with the 10th layer of processing, which is about how many "processing stages" Serre, Oliva, & Poggio claim are in the ventral stream. I find this whole argument misleading, given that the specific layer depth that correlates with humans is going to vary monotonically with the total depth of the network. If you consider their result that the highest correlation is always at around 70% of the total depth, it seems reasonable to just choose a network that has the correct number of total layers to match the Serre et al.'s findings. Also, they claim that they use VGG-16 because it is the best performing network, but that is not true according to Figure 3a. VGG-19 performs best out of the box, only after fine-tuning VGG-16 do they do the best. They do not comment on how well a fine-tuned VGG-19 net would do.