Do Convnets Learn Correspondence?
Convolutional neural nets (convnets) trained from massive labeled datasets [1] have substantially improved the state-of-the-art in image classification [2] and object detection [3]. However, visual understanding requires establishing correspondence on a finer level than object category. Given their large pooling regions and training from whole-image labels, it is not clear that convnets derive their success from an accurate correspondence model which could be used for precise localization. In this paper, we study the effectiveness of convnet activation features for tasks requiring correspondence. We present evidence that convnet features localize at a much finer scale than their receptive field sizes, that they can be used to perform intraclass aligment as well as conventional hand-engineered features, and that they outperform conventional features in keypoint prediction on objects from PASCAL VOC 2011 [4].
The paper analyzes the degree to which features in CNNs identify object or part locations in images, even though they may not be trained for localization. There are three sets of tests. The first is to find nearest neighbors for patches based on their CNN activations; the second warps an image to its nearest neighbor patches, and the third transfers keypoints. The results of these experiments indicate that CNN features could be a replacement for SIFT in computer vision tasks. Overall, the topic and results here generally justify publication of the paper. The paper addresses a very interesting and relevant question that could definitely have impact on the vision literature.There are two major problems with the paper.The first problem is that most of the results are very unconvincing as they are presented. * The results in Figure 1 are not trustworthy. It is easy for a human viewer to be mislead by such a figure, i.e., to infer that correspondences are correct even when they are not, like in a Photomosaic (http://www.photomosaic.com/). More needs to be done to verify that the claimed correctness of correspondences is actually correct, like showing a visualization of class/part labels. Also, it doesn't help that the resolution of this figure is terrible; it is blurry even at print resolution. * I do not understand the results in Figure 2 at all. It is unclear what is meant by receptive field here: I would assume it's the domain of the filters, but pixels are shown here.* The results in Figure 3 are unpersuasive and ambiguous, but the scores in Table 2 seem fine to support the point that the results are comparable to SIFT (though not better). * The results in Figure 6 and Table 4 are persuasive to me, but I'm not an expert in keypoint prediction, so I can't really judge the significance of this result or the experimental setup.Second, the paper is not clearly written; it reads like a class project report, in which many terms and acronyms are used without definition. In many cases one can make educated guesses (and being familiar with the literature and the Caffe implementation helps), but one should not have to guess what is in the authors' heads. Examples: it is never defined how CNNs produce "features" (presumably the vector of filter outputs); the acronym "rfs" is never defined, nor is the receptive field that it refers to; in Section 3, the problem statement and inputs and outputs are never defined, nor is the source or target image. It's not actually clear to me what is meant by receptive field (see above).One other comment: an impact of this paper is that it might encourage vision researchers to replace SIFT with CNN features. It would be helpful to hear about the practicalities of doing so, in particular the memory and computation time requirements of doing so.