We consider learning based methods for visual localization that do not require the construction of explicit maps in the form of point clouds or voxels. The goal is to learn an implicit representation of the environment at a higher, more abstract level, for instance that of objects. We propose to use a generative approach based on Generative Query Networks (GQNs, Eslami et al., 2018), asking the following questions: 1) Can GQN capture more complex scenes than those it was originally demonstrated on? 2) Can GQN be used for localization in those scenes? To study this approach we consider procedurally generated Minecraft worlds, for which we can generate images of complex 3D scenes along with camera pose coordinates. We first show that GQNs, enhanced with a novel attention mechanism can capture the structure of 3D scenes in Minecraft, as evidenced by their samples. We then apply the models to the localization problem, comparing the results to a discriminative baseline, and comparing the ways each approach captures the task uncertainty.
**Summary of the paper**
This paper studies the problem of visual re-localization, where we are interested in estimating the camera pose of a new image from a set of source images and their camera poses. Instead of explicitly designing the structure of a map of 3D scenes (e.g. occupancy grids or point clouds), the paper proposes implicitly learning an abstract map representation. Specifically, the paper proposes a generative method based on Generative Query Networks (GQNs) augmented with an attention mechanism. The authors apply this model to the visual re-localization problem. To train and test the proposed model, the authors introduce the Minecraft random walk dataset, which consists of images and their camera poses extracted from randomly generated trajectories in the Minecraft environment. The proposed model is compared against a discriminative counterpart, which is trained to directly predict the target camera pose and achieves better MSE.
**Clarity**
Above average
**Significance**
Below Average
**Detailed comments**
_Paper Strengths_
- The idea of leveraging generative models' knowledge of "maps" to perform visual localization is interesting. This gives learning frameworks the flexibility of building a latent representaiton of maps which may yield better performance instead of being restricted to a pre-defined representations.  
- The paper is very well-written and easy to follow. 
- The authors did a good job presenting the proposed methods. The descriptions and formulations are clear. Both Figure 2 and Figure 3 are helpful for understanding the GQNs and the proposed attention mechanism.
- The patch dictionary for the attention mechanism seems effective especially when dealing with a set of context images capturing the same scene.
- The authors are honest about the limitations of the proposed framework compared to classic approaches.
- The visualization of results are clear. Particularly, Figure 5 and Figure 7 give easily interpretable representations of the results.
_Paper Weaknesses_
- Implicitly learning a map of the scene is mentioned as a strength in the paper, but this comes at the high cost of interpretability. Without an explicit map representation, it is difficult to understand the failure cases - does the model not understand the 3D scene well or does the model have a hard time accurately predicting camera poses?
- Minecraft is an interesting environment for proof of concept, but lacks much of the subtlety of the real world.
- Building a framework that is able to perform the localization task from real-world scenes is more interesting. Learning generative models of real-world scenes is known to be difficult, which makes this framework impractical. There are google streetview and indoor datasets authors can try to utilize.
- The aforementioned point is supported by the fact that the localization performance of the proposed model on real-world scenes is missing.  
- The reviewer does not find enough novelty from the proposed model, which is an iterative improvement on GQNs.
- The paper only compares the proposed model against its discriminative couterpart, which is not sufficient. While the authors strongly argue that exploiting the proposed implicitly representations of scenes is more beneficial than utilizing the pre-defined explicit representations, the only baseline is using the same implicit representations. Although the reviewer is aware of that this model does not use complete video sequences, benchmarking against a visual monocular SLAM algorithm, like LSD-SLAM [1], would contextualize the claim.
- Why quantize the discriminative model's output? This de-correlates nearby pose values. The paper could benefit from an explanation of not using a straightforward regression over pose variables.
- Overall, the reviewer does not find enough novelty from any aspects except the idea of utilizing a generative model for visual localization with implocitly learned maps, which is not fully demonstrated in the experiment section (i.e. not compare to baselines using explicit maps). 
- A differentiating factor for this paper could be tackling one of the open problems remaining in SLAM as identified in [2], like lifelong learning or semantic mapping.
_Reproducibility_
- Given the clear description in the main paper and the details provided in the appendix, the reviewer believes reproducing the results is possible. 
_Conclusion_
- Overall, the reviewer believes this paper is well presented and reproducible. However, the paper does not propose to solve a novel problem, nor does it present a very novel method. Although the idea of using existing generative networks for localization is interesting, the paper misses important baselines relying on explicit map representation and is not sufficiently convincing. Moreover, requiring a generative model significantly limits the possibility of utilizing the proposed model for real-world applications. While the paper does present a new dataset built in Minecraft which is suitable for demonstrating the strengths of the proposed method, the reviewer does not find this significant. Therefore, the reviewer recommends a rejection.
_Reference_
[1] Engel, Jakob, Thomas Schöps, and Daniel Cremers. "LSD-SLAM: Large-scale direct monocular SLAM." European Conference on Computer Vision. Springer, Cham, 2014.
[2] Cadena, Cesar, et al. "Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age." IEEE Transactions on Robotics 32.6 (2016): 1309-1332.