Neighbourhood Consensus Networks
We address the problem of finding reliable dense correspondences between a pair of images. This is a challenging task due to strong appearance differences between the corresponding scene elements and ambiguities generated by repetitive patterns. The contributions of this work are threefold. First, inspired by the classic idea of disambiguating feature matches using semi-local constraints, we develop an end-to-end trainable convolutional neural network architecture that identifies sets of spatially consistent matches by analyzing neighbourhood consensus patterns in the 4D space of all possible correspondences between a pair of images without the need for a global geometric model. Second, we demonstrate that the model can be trained effectively from weak supervision in the form of matching and non-matching image pairs without the need for costly manual annotation of point to point correspondences. Third, we show the proposed neighbourhood consensus network can be applied to a range of matching tasks including both category-and instance-level matching, obtaining the state-of-the-art results on the PF Pascal dataset and the InLoc indoor visual localization benchmark.
Paper overview: The authors propose a novel, end-to-end trainable deep architecture for finding dense correspondences between pairs of images. First, dense features are extracted from both images using a 2D CNN. Then, an exhaustive cross-correlation is performed between the feature maps to construct a 4D matching hyper-volume, which is fed to a "neighbourhood consensus network". This is, in essence, a 4D convolutional architecture which evaluates spatially local subsets of correspondences in order to detect latent consensus patterns, in a way reminiscent of traditional semi-local matching approaches. The final output of the network is a probability distribution over the set of dense matches between the images. The network is trained in a weakly-supervised manner, by processing pairs of matching / non matching images, without any explicit annotation on correspondences.  Comments: The paper is very well written, clear and easy to follow. The authors manage to convincingly communicate the intuition motivating their approach, and the technical sections are easy to understand. The problem being tackled is a fundamental one, and the proposed approach is interesting and novel (but see question #2 below). The experimental evaluation is quite convincing, even if I would have liked to see some quantitative results on the accuracy of instance-level matching, in addition to the final localization results when the proposed method is used as a component of InLoc. Over all this is a very good submission, but there are two main points I would like the authors to clarify:  1) As far as I can understand, the weakly-supervised loss described in section 3.5 doesn't impose any kind of "physically meaningful" constraint on the matches. The network could learn to select _any_ set of correspondences, as long as the soft and hard assignments for the positive image pairs are consistent with each other, even if these correspondences do not make any sense given the content of the two images. In essence, what is being optimized here is close to an entropy loss over the matching probability distributions. Is this the case? Have you ever observed any "drifting" behavior during training that could be due to this?  2) The proposed approach shares some similarities with recent works on stereo matching such as [1], i.e. it works by processing a matching cost volume through a CNN. Can the authors comment on this?  [1] Kendall, Alex, et al. "End-to-end learning of geometry and context for deep stereo regression." CoRR, vol. abs/1703.04309 (2017)