Recurrent Transformer Networks for Semantic Correspondence
We present recurrent transformer networks (RTNs) for obtaining dense correspondences between semantically similar images. Our networks accomplish this through an iterative process of estimating spatial transformations between the input images and using these transformations to generate aligned convolutional activations. By directly estimating the transformations between an image pair, rather than employing spatial transformer networks to independently normalize each individual image, we show that greater accuracy can be achieved. This process is conducted in a recursive manner to refine both the transformation estimates and the feature representations. In addition, a technique is presented for weakly-supervised training of RTNs that is based on a proposed classification loss. With RTNs, state-of-the-art performance is attained on several benchmarks for semantic correspondence.
The paper proposes an recurrent iterative refinement approach for weakly supervised semantic correspondence learning between similar objects. The idea is to predict a flow field transformation for each pixel, such that after transforming the image the source and target images are aligned. This paper builds on the weak-supervision similar to [18] and [29] but predicts the flow, and adds an iterative component where in each step the network outputs the residual to that brings the source and the transformed target images closer.  The objective is similar to the concurrent work of [29] (but with negatives) where the idea that transformed pixel j should have the highest feature correlation with the feature at the pixel it got transformed to than any of its immediate neighbors. The paper also proposes a dilated search window for this neighbor radius, where the search window shrinks as the iteration continues such that more finer alignment may be achieved.  The paper evaluates against standard benchmarks, improves upon recent state of the art methods, and obtains visually pleasant results.  Pros of this paper is that: - Results look good - Doing matching in an iterative style makes sense - Progressively shrinking search window makes sense - In a sense it's a simple approach that brings together a lot of the recent   advancements in this field. - Flow rather than affine or TPS allows for finer detail matching.  Weaknesses are: In terms of originality & novelty, while the idea to do iteration has yet to be tried in image matching, this idea has been demonstarted in various other applications:   - Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, Jitendra Malik, "Human   Pose Estimation with Iterative Error Feedback", CVPR 2016   - Markus Oberweger, Paul Wohlhart, Vincent Lepetit, "Training a feedback loop     for hand pose estimation" ICCV 2015   - Piotr Dollar, Peter Wlinder, Pietro Perona, "Cascaded pose regression", CVPR     2010  The paper should be more generous in citing these papers.  In an essence this paper could be seen as a combination of [18] and [29] but with flow and iteration + other tricks. Therefore it's not as original, however, [29] is concurrent and given the quality of results I'm inclined towards acceptance. At the same time, this kind of paper might be more suitable for vision conferences like CVPR/ICCV/ECCV.  Authors might want to cite these other related works: - Thewlis et al. Unsupervised learning of object frames by dense equivariant   image labelling. NIPS 2017 Does not learn matching transformation but learns a matchable features, also from synthetic deformations - Kanazawa et al. "WarpNet: Weakly Supervised Matching for Single-view   Reconstruction", one of the first papers to solve for TPS to match two images, trained in a self-supervised manner through synthetic transformation (line 123) and also they demonstrate the matching results for single-view 3D application, relevant to the conclusion of this paper.  Post rebuttal:  I stand by my original rating.