In geometric computer vision applications, multi-image feature matching gives more accurate and robust solutions compared to simple two-image matching. In this work, we formulate multi-image matching as a graph embedding problem, then use a Graph Neural Network to learn an appropriate embedding function for aligning image features. We use cycle consistency to train our network in an unsupervised fashion, since ground truth correspondence can be difficult or expensive to acquire. Geometric consistency losses are added to aid training, though unlike optimization based methods no geometric information is necessary at inference time. To the best of our knowledge, no other works have used graph neural networks for multi-image feature matching. Our experiments show that our method is competitive with other optimization based approaches.
The paper presents graph neural network approach for learning multi-view feature similarity. 
The input data is local feature descriptors (SIFT), keypoint location, orientation and scale. The objective is to learn embedding such that features corresponding to the same 3d location will have similar embeddings, while different - far away. Such embedding distance matrix is called "feature universe" in the paper. 
Instead of ground truth correspondence matrix, authors use smth called "noisy adjacency matrix", although it was not clear to me, what does it mean precisely. Loss is also augmented with epipolar distance reprojection error. 
Training and testing is performed on Rome16k dataset. 
Overall I find paper hard to follow and weak on experiment side. Comments and questions:
 1) There is no proper (or any) traditional baseline, which is: match SIFT features, apply cross-consistency and SNN ratio thresholds, run RANSAC, throw away all the inconsistent things. See COLMAP, Bundler or any other Visual SfM/MVS pipeline. Moreover, Hartman et.al. method is cited, but not compared to, because it needs "3d reconstruction" for supervision. Here is my second objection. 
 
 2) Paper states that the method is unsupervised. Yet, it is based on known scene geometry (R and t) , which is typically obtained via 3d reconstruction pipeline. Once you do it,  you actually have ground truth correspondences, a lot of them. I don`t understand, why not use them. 
 
 3) Experimental validation is also weird. Method was trained on Rome16k and tested on it as well. No other datasets were used besides Graffity sequence (see below). The metric is L1 norm on adjecency matrix instead of some.
 
 4) Regarding "feature universe". It is clear, that one cannot fit (or can?) the full n_features x n_3d_points matrix into GPU memory, as it should be super huge matrix. No details were given on how such problems are tackled.  
 5) The paper, unfortunately, has a number of side claims, which are false and actually irrelevant to the paper core. Just to list a few: "deep learning has revolutionized how image features are computed (Yi et al., 2016)." It has not. SIFT is still quite the gold standard:  (To Learn or Not to Learn: Visual Localization from Essential Matrices, https://arxiv.org/abs/1908.01293, https://image-matching-workshop.github.io), besides that cited LIFT method was not used in practice because of being super slow.
 - "More fundamentally, deep neural networks need large amounts of labeled data to train. In the case of multi-image feature matching, one would need hand-labeled point correspondences between images, which can be difficult and expensive to obtain.".
  Nobody hand-labels correspondences. Instead, one uses runs 3d reconstruction pipeline with densification like COLMAP to obtain dense depth map, what where one can get multiview correspondences (e.g., MegaDepth: Learning Single-View Depth Prediction from Internet Photos https://arxiv.org/abs/1804.00607) 
  - "Typically putative correspondences are matched probabilistically, meaning a feature in one image matches to many features
in another. The ambiguity in the matches could come from repeated structures in the scene, insuffi-
ciently informative low-level feature descriptors, or just an error in the matching algorithm. Filtering
out these noisy matches is our primary learning goal.". Typically, one-to-many matches are just thrown away (e.g from Bundler SfM paper "Modeling the World from Internet Photo Collections", Sec.4.2: "If a track contains more than one keypoint in the same image, it is deemed inconsistent.
We keep consistent tracks containing at least two keypoints for the next phase of the reconstruction procedure").
 6) Citations are sometimes weird. E.g. part of OxfordAffine dataset (http://www.robots.ox.ac.uk/~vgg/research/affine/)  is referred as Graffity without any reference at all to the dataset itself (???), but with references to two irrelevant works which are testing on it. Why benchmark sycle consistency of such a small dataset of a flat surfaces? Then one could use Fountain sequence, at least, which has some non-planar structures on it. 
 
 
 Minor Comments:
 
  - Table 1 is hardly readable because of scientific notation used.
****
After rebuttal update.
I am increasing my score a bit, but still think that paper is not is the shape for publishing. Method itself might be good, but evaluation is still bad, and what is worse - authors haven`t even tried to improve it.
>At the end of the review period it will be revealed that we have 20 years of publication history in structure from motion and visual odometry.  
This, unfortunately, does not help the current paper.
>We do in fact test on data not in our training set: we never trained on Graffiti. 
Graffity is still only 6 (six!!!) images, which are related by homography. One could show results, at least for HSequences (118 * 6) images or add other dataset. Since, it is not training, it could be done quite fast.
> However, the focus of our paper is a novel feature representation, the formulation using a GNN framework, and the self-supervised losses. Overall, our paper is a novel approach to learning feature representations, a topic of great interest to the ICLR community, rather than a new structure from motion system that has to prove its superior performance over the current state of the art.
I completely agree with it and would be happy to accept such paper to any kind of workshop.