This work explores the visual explanation for deep metric learning and its applications. As an important problem for learning representation, metric learning has attracted much attention recently, while the interpretation of such model is not as well studied as classification. To this end, we propose an intuitive idea to show where contributes the most to the overall similarity of two input images by decomposing the final activation. Instead of only providing the overall activation map of each image, we propose to generate point-to-point activation intensity between two images so that the relationship between different regions is uncovered. We show that the proposed framework can be directly deployed to a large range of metric learning applications and provides valuable information for understanding the model. Furthermore, our experiments show its effectiveness on two potential applications, i.e. cross-view pattern discovery and interactive retrieval.
= Summary 
This paper presents a simple method that draws visual attention of deep embedding networks for metric learning. It basically follows the class attention mapping strategy based on global pooling operation [Zhou et al., CVPR 2016], but extends the original version to point-specific attention which is novel and enable interesting applications on image retrieval. In addition, the proposed method seems also independent of the loss function used for metric learning, thus it can be applied to most of existing deep embedding networks to understand their behaviors in a qualitative manner. 
= Decision
My current decision is officially "weak reject" but "borderline" in my mind. The major concern of mine is its weaknesses in clarity and technical novelty. However, I still believe this submission is valuable since it addresses a relatively new and timely topic, the proposed method is simple yet effective, and the applications of point-specific attention (i.e., "cross-view pattern discovery" and "interactive retrieval") are all interesting and practically useful. If the clarity issues are all clearly addressed, I would upgrade my rating. 
= Comments
[Pros]
1) The motivation and implementation of the point-specific attention are convincing. 
2) The point-specific attention enables not only image-to-image retrieval, but also more elaborate understanding about a pair of images and their similarity.
3) The proposed technique is model-agnostic and loss-agnostic, thus can be applied to most of existing deep embedding networks for image retrieval. Also, the proposed technique does not degrade the retrieval performance.
4) The proposed technique is simple yet effective in multiple applications, most of which are practically useful and have great impact. For example, the weakly supervised localization is an essential step towards many weakly supervised approaches for higher-level recognition tasks like semantic segmentation, and the interactive retrieval will allow us to build more realistic and useful image retrieval systems.
[Cons]
1) The proposed technique itself is not new but a straightforward extension of an existing work [Zhou et al., CVPR 2016].
2) The manuscript is not crystal clear.
- The name of the proposed point-specific attention (i.e., "Partial attention") is misleading.
- The way to compute the point-specific attention map is not clearly described in Section 3.
- It is hard to understand the contents in Figure 6 as they are not clearly illustrated. 
- The experimental and implementation details of the last two applications are not given. For example: (cross-view pattern discovery) how to compute the angle error, and how much the proposed technique is sensitive to the position selected on the query image, (interactive retrieval) how to compute the similarity between images given a specific region of interest on query.
3) More qualitative results on the last two applications should be presented, even in an appendix, to convince future readers. Especially, in the case of "interactive retrieval", more results are demanded as quantitative performance analysis seems not straightforward.
= Post-rebuttal review
The rebuttal resolves my major concerns and the manuscript has been carefully revised accordingly. So as I promised in my original review, I upgrade the score to weak accept.