Matching neural paths: transfer from recognition to correspondence search
Many machine learning tasks require finding per-part correspondences between objects. In this work we focus on low-level correspondences -a highly ambiguous matching problem. We propose to use a hierarchical semantic representation of the objects, coming from a convolutional neural network, to solve this ambiguity. Training it for low-level correspondence prediction directly might not be an option in some domains where the ground-truth correspondences are hard to obtain. We show how transfer from recognition can be used to avoid such training. Our idea is to mark parts as "matching" if their features are close to each other at all the levels of convolutional feature hierarchy (neural paths). Although the overall number of such paths is exponential in the number of layers, we propose a polynomial algorithm for aggregating all of them in a single backward pass. The empirical validation is done on the task of stereo correspondence and demonstrates that we achieve competitive results among the methods which do not use labeled target domain data.
This paper presents a method for determining spatial correspondences between two images using a pre-trained convolutional network, without needing to train on the specific correspondence task.  Given two images, the matching similarity at each pixel location is defined as the sum total similarity between all pairs of possible paths through network, starting at each location and its corresponding offset location in the other image.  For a single offset value, this results in a recurrence relation dependent on the next layer up, which enables a dynamic-programming-like algorithm to compute all costs in linear time using a single backwards pass.  In this way, a multi-layer matching is performed for each offset, accounting for a hierarchy of scales and features.
This algorithm is applied to the task of stereo matching, where it achieves performance slightly lower than SOA supervised methods that require retraining, but better than a baseline computing correlation between stacked feature maps.
This idea is an interesting way to use a pretrained network for additional uses.  However, I think it could be further explored and evaluated, and perhaps more simply explained.  In particular, I would like to see whether it can be applied to other dense matching tasks, e.g. video optical flow, and compared to other dense correspondence methods, such as Deep Flow and SIFT Flow (in both speed and accuracy).  In addition, the notation seems a little overly complex and can be hard to follow (though I was able to understand it) --- I think it would be good to try simplifying this if possible.
Overall, I think this is an interesting method, but feel it could be further explored for more image correspondence applications.
Additional detail:  Twice the paper says the derivation of the algorithm makes use of the "associative" property, but this is actually the distributive property (not associative).