Capturing Human Category Representations by Sampling in Deep Feature Spaces
Understanding how people represent categories is a core problem in cognitive science, with the flexibility of human learning remaining a gold standard to which modern artificial intelligence and machine learning aspire. Decades of psychological research have yielded a variety of formal theories of categories, yet validating these theories with naturalistic stimuli remains a challenge. The problem is that human category representations cannot be directly observed and running informative experiments with naturalistic stimuli such as images requires having a workable representation of these stimuli. Deep neural networks have recently been successful in a range of computer vision tasks and provide a way to represent the features of images. In this paper, we introduce a method for estimating the structure of human categories that draws on ideas from both cognitive science and machine learning, blending human-based algorithms with state-of-the-art deep representation learners. We provide qualitative and quantitative results as a proof of concept for the feasibility of the method. Samples drawn from human distributions rival the quality of current state-of-the-art generative models and outperform alternative methods for estimating the structure of human categories.
 The idea of using MCMCP with GANs is well-motivated and well-presented in the paper, and the approach is new as far as I know. Figures 3 and 5 are convincing evidence that MCMCP compares favorably to direct sampling of the GAN feature space using the classification images approach. However, as discussed in the introduction, the reason an efficient sampling method might be interesting would be to provide insight on the components of perception. On these insights, the paper felt incomplete. For example, it was not investigated whether the method identifies classification features that generalize. The faces experiment is similar to previous work done by Martin (2011) and Kontsevich (2004) but unlike that previous work does not investgiate whether classification features have been identified that can be added to an arbitrary image to change the attribute "happy vs sad" or "male vs female". Similarly, the second experiment in Table 1 compares classification accuracy between different sampling methods, but it does not provide any comparison as done in Vondrick (2015) to a classifier trained in a conventional way (such as an SVM), so it is difficult to discern whether the learned distributions are informative. Finally, the effect of choosing GAN features vs a more "naive" feature space is not explored in detail. For example, the GAN is trained on an image data set with many birds and cars but not many fire hydrants. Is the method giving us a picture of this data set?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Capturing Human Category Representations by Sampling in Deep Feature Spaces
Understanding how people represent categories is a core problem in cognitive science, with the flexibility of human learning remaining a gold standard to which modern artificial intelligence and machine learning aspire. Decades of psychological research have yielded a variety of formal theories of categories, yet validating these theories with naturalistic stimuli remains a challenge. The problem is that human category representations cannot be directly observed and running informative experiments with naturalistic stimuli such as images requires having a workable representation of these stimuli. Deep neural networks have recently been successful in a range of computer vision tasks and provide a way to represent the features of images. In this paper, we introduce a method for estimating the structure of human categories that draws on ideas from both cognitive science and machine learning, blending human-based algorithms with state-of-the-art deep representation learners. We provide qualitative and quantitative results as a proof of concept for the feasibility of the method. Samples drawn from human distributions rival the quality of current state-of-the-art generative models and outperform alternative methods for estimating the structure of human categories.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The idea of using MCMCP with GANs is well-motivated and well-presented in the paper, and the approach is new as far as I know. Figures 3 and 5 are convincing evidence that MCMCP compares favorably to direct sampling of the GAN feature space using the classification images approach. However, as discussed in the introduction, the reason an efficient sampling method might be interesting would be to provide insight on the components of perception. On these insights, the paper felt incomplete. For example, it was not investigated whether the method identifies classification features that generalize. The faces experiment is similar to previous work done by Martin (2011) and Kontsevich (2004) but unlike that previous work does not investgiate whether classification features have been identified that can be added to an arbitrary image to change the attribute "happy vs sad" or "male vs female". Similarly, the second experiment in Table 1 compares classification accuracy between different sampling methods, but it does not provide any comparison as done in Vondrick (2015) to a classifier trained in a conventional way (such as an SVM), so it is difficult to discern whether the learned distributions are informative. Finally, the effect of choosing GAN features vs a more "naive" feature space is not explored in detail. For example, the GAN is trained on an image data set with many birds and cars but not many fire hydrants. Is the method giving us a picture of this data set?
assistant
 The idea of using MCMCP with GANs is well-motivated and well-presented in the paper, and the approach is new as far as I know. Figures 3 and 5 are convincing evidence that MCMCP compares favorably to direct sampling of the GAN feature space using the classification images approach. However, as discussed in the introduction, the reason an efficient sampling method might be interesting would be to provide insight on the components of perception. On these insights, the paper felt incomplete. For example, it was not investigated whether the method identifies classification features that generalize. The faces experiment is similar to previous work done by Martin (2011) and Kontsevich (2004) but unlike that previous work does not investgiate whether classification features have been identified that can be added to an arbitrary image to change the attribute "happy vs sad" or "male vs female". Similarly, the second experiment in Table 1 compares classification accuracy between different sampling methods, but it does not provide any comparison as done in Vondrick (2015) to a classifier trained in a conventional way (such as an SVM), so it is difficult to discern whether the learned distributions are informative. Finally, the effect of choosing GAN features vs a more "naive" feature space is not explored in detail. For example, the GAN is trained on an image data set with many birds and cars but not many fire hydrants. Is the method giving us a picture of this data set?