Universal Correspondence Network
We present a deep learning framework for accurate visual correspondences and demonstrate its effectiveness for both geometric and semantic matching, spanning across rigid motions to intra-class shape or appearance variations. In contrast to previous CNN-based approaches that optimize a surrogate patch similarity objective, we use deep metric learning to directly learn a feature space that preserves either geometric or semantic similarity. Our fully convolutional architecture, along with a novel correspondence contrastive loss allows faster training by effective reuse of computations, accurate gradient computation through the use of thousands of examples per image pair and faster testing with O(n) feed forward passes for n keypoints, instead of O(n 2 ) for typical patch similarity methods. We propose a convolutional spatial transformer to mimic patch normalization in traditional features like SIFT, which is shown to dramatically boost accuracy for semantic correspondences across intra-class shape variations. Extensive experiments on KITTI, PASCAL, and CUB-2011 datasets demonstrate the significant advantages of our features over prior works that use either hand-constructed or learned features.
for example, SIFT or SURF for sparse structure from motion, DAISY or DSP for dense matching, SIFT Flow or FlowWeb for semantic matching. The Universal Correspondence Network accurately and efficiently learns a metric space for geometric correspondences, dense trajectories or semantic correspondences. explicit interpretation. In addition, patch similarity is inherently inefficient, since features have to be extracted even for overlapping regions within patches. Further, it requires O(n 2 ) feed-forward passes to compare each of n patches with n other patches in a different image.
In contrast, we present the Universal Correspondence Network (UCN), a CNN-based generic discriminative framework that learns both geometric and semantic visual correspondences. Unlike many previous CNNs for patch similarity, we use deep metric learning to directly learn the mapping, or feature, that preserves similarity (either geometric or semantic) for generic correspondences. The mapping is, thus, invariant to projective transformations, intra-class shape or appearance variations, or any other variations that are irrelevant to the considered similarity. We propose a novel correspondence contrastive loss that allows faster training by efficiently sharing computations and effectively encoding neighborhood relations in feature space. At test time, correspondence reduces to a nearest neighbor search in feature space, which is more efficient than evaluating pairwise patch similarities.
The UCN is fully convolutional, allowing efficient generation of dense features. We propose an on-the-fly active hard-negative mining strategy for faster training. In addition, we propose a novel adaptation of the spatial transformer [13], called the convolutional spatial transformer, desgined to make our features invariant to particular families of transformations. By learning the optimal feature space that compensates for affine transformations, the convolutional spatial transformer imparts the ability to mimic patch normalization of descriptors such as SIFT. Figure 1 illustrates our framework.
The capabilities of UCN are compared to a few important prior approaches in Table 1. Empirically, the correspondences obtained from the UCN are denser and more accurate than most prior approaches specialized for a particular task. We demonstrate this experimentally by showing state-of-the-art performances on sparse SFM on KITTI, as well as dense geometric or semantic correspondences on both rigid and non-rigid bodies in KITTI, PASCAL and CUB datasets.
To summarize, we propose a novel end-to-end system that optimizes a general correspondence objective, independent of domain, with the following main contributions:
• Deep metric learning with an efficient correspondence constrastive loss for learning a feature representation that is optimized for the given correspondence task.
• Fully convolutional network for dense and efficient feature extraction, along with fast active hard negative mining.
• Fully convolutional spatial transformer for patch normalization.
• State-of-the-art correspondences across sparse SFM, dense matching and semantic matching, encompassing rigid bodies, non-rigid bodies and intra-class shape or appearance variations.
Unlike previous patch similarity based methods, in this work the authors proposed a new model that discover visual correspondences using dense feature vector, correspondence contrastive loss function (a combination of mean squared loss and margin-based ranking loss), and CNN Spatial Transformer. The proposed method can successfully capture both geometric correspondence and semantic correspondence.
In this paper the authors proposed a universal correspondence network using CNN and pair-wise ranking loss. Overall this paper is well written and easy to follow. Below are the detailed review.  1. Equation(1) is the loss function used in the proposed model and s_i/(1-s_i) is acted as the identity function to select which part of the loss function will be used. In Line 121 the authors said they denote s = 0 for positive pairs and s=1 for negative pairs. In such way positive pairs will be trained by margin-based ranking loss and negative pairs will be trained by mean squared loss. This conflicts to Line 132 which claims that the mean squared loss is used for minimizing the distance between positive pairs and ranking loss is used to make sure negative pairs are at least m distance away.  2. In this work, all the key points are pre-selected so is it possible to run this algorithm without given any key points and let the model decide this? Basically this would become given a dense feature vector from left image, rank all vectors in the right image. Does training with hard negative pairs affect the performance in that case since a lot of vectors will not be updated during the training process?  3. The authors use a spatial transformer to mimic path normalization and show that by adding this the performance of the model increases. Does this differ from shortcuts? If so what is the advantage of spatial transformer over shortcut?  4. Figure 5 shows the Accuracy of PCK performance but no precision and recall. If I understand correctly, for a certain pixel threshold k, does the accuracy equal to #(actual correspondent pixels) / k? If so it would be interesting to see how the recall shape looks like.