An important goal in deep learning is to learn versatile, high-level feature representations of input data. However, standard networks' representations seem to possess shortcomings that, as we illustrate, prevent them from fully realizing this goal. In this work, we show that robust optimization can be re-cast as a tool for enforcing priors on the features learned by deep neural networks. It turns out that representations learned by robust models address the aforementioned shortcomings and make significant progress towards learning a high-level encoding of inputs. In particular, these representations are approximately invertible, while allowing for direct visualization and manipulation of salient input features. More broadly, our results indicate adversarial robustness as a promising avenue for improving learned representations.
###  Summary
1. The paper proposes robustness to small adversarial perturbations as a prior when learning representations. 
2. It demonstrates that representations that satisfy such a prior have non-trivial properties -- they are easier to visualize, are invertible (i.e. optimizing an input that produces the desired activation leads to reasonable images), and allows for direct manipulation of input features (by changing a feature in the representation space and then optimizing the image to satisfy this new representation.)
### Non-blind review 
THIS IS NOT A BLIND REVIEW
Reviewing this paper reminded me of a recent NeurIPS paper I read. 
I went back to that NeurIPS paper (to better compare the similarities and differences) only to found out:
1- The NeurIPS19 paper cites an earlier arxiv version of this paper as an inspiration for its approach. 
2- It is from the exact same authors. 
This, unfortunately, means I know who the authors are (however, there is no conflict of interest). 
More importantly, this paper is too similar to the NeurIPS paper and It's hard to review without taking into account the NeurIPS paper. In this review, I will treat the said NeurIPS19 paper as published work, and evaluate if this work adds more to the discourse. (I've refrained from naming the neurips paper so the anonymity is maintained for other reviewers; the authors, I presume, would immediately know which paper I'm referring to). 
### Decisions with reasons
Even though I think the idea introduced in this paper is interesting, I would argue for rejecting this paper for the simple reason: It doesn't add much to the existing discourse. 
Using the proposed framework (i.e. learning robust representations), it demonstrates two phenomena.
First, it shows that robust models allow feature inversion. Second, it shows that it's easily possible to directly visualize and manipulate features for such a model. (Both of these are achieved using the same idea: treating input to the model as parameterized, and optimizing for a target activation)
These are interesting observations and show that robust models learn features that rely on salient parts of the input image. However, the NeurIPS19 paper shows this even more cleary. 
As a result, I'm not convinced that demonstrating the same phenomena with different examples is sufficient for this to be a standalone paper. (Perhaps the two papers could have been one single paper). 
### Questions 
What is the rationale behind dividing examples showing robust models rely on salient parts of input into two papers? Is there a semantic meaning to the grouping i.e. showing feature inversion, feature manipulation, and visualization in one paper and Generation, inpainting, translation, etc in another?
If I understand correctly, all of these examples exist because the robustly learned representation relies on the salient parts of the input and not on the non-robust features. If that is the case, it makes more sense to show all of these examples in a single paper. 
### Update after Author's response
Since the authors have added and discussed the pertinent NeurIPS paper in this submission, I'm updating my score. 
I still think that the two papers are more similar than they might seem (See Re: Response 3 for more details). 
### Update 2 
I pointed out the similarities between the three contributions in this paper and the NeurIPS paper in "Re: Response #3" below. The authors replied to my concerns. I'm summarizing the author's position to my concerns followed by my response. 
#### Author's Position
The authors agreed that features manipulation and feature visualization is similar, but pointed out that the chain of dependency is this paper -> NeurIPS paper and not the other way around. They mentioned that the NeurIPS paper cites this paper and acknowledges this. Moreover, they argued that even if we consider NeurIPS paper to be prior work, feature visualization is explored in much more detail in this paper. 
#### Response 
I think the direction of the chain of dependency is not that important since neither paper clearly builds on top of the other. The NeurIPS paper is published work now, and it makes sense to consider it prior work (Especially since it is from the same authors). 
Moreover, during the NeurIPS review period, the authors did not cite this paper; they only added the citation in the camera-ready version. This means that during the NeurIPS review period, they did, in fact,  take credit for the ideas used in feature painting. (The authors mention that they somehow did not, and just stated the method and showed the pictorial result in the NeurIPS paper. However, I don't see how it is possible to present a method and a pictorial result without citing other work and not take credit for the method and result.) 
I would agree with the authors that this paper does go into more detail for feature visualization. More specifically, this paper also looks at visualizing individual features in the representation (The NeurIPS feature painting restricts the visualization using a mask) and demonstrates that the same feature can be used to visualize similar semantic concepts (such as red limbs) on multiple images. This is definitely interesting, but still very related to the feature-painting result. It would have made more sense to include these feature visualization results in the NeurIPS paper instead of adding them in a separate paper.
#### Author's Position 2
They disagreed that feature inversion (this paper) is similar to image generation (NeurIPS paper). I did acknowledge in my initial response that feature inversion is slightly more general than image generation, however, the authors suggest that they are completely different."
#### Response 
I think representation inversion is more similar to generation than it might seem. Representation for an in-distribution image would correspond to a class with high probability. Maximizing a class probability would indirectly optimize for a representation (Say R_0) that maximizes that class probability. Image generation, as presented in NeurIPS paper, can be seen as inverting R_0.
 
Moreover, the qualitative results for feature inversion, as presented in this paper, are not extra-ordinary. In the majority of the inverted images, I can not classify the inverted image correctly. That shows the model is still not paying attention to the correct aspects of the input to do classification. As a result, this paper certainly does not solve the feature inversion problem (Ideally, inverted features would highlight parts of the input necessary for making predictions and ignore other parts. Robust models, on the other hand, seem to be uniformly retaining all information of the image including the background and not highlighting the parts important for making predictions. As a result, many inverted images can not be classified by humans). 
#### My current position 
At the end of the day, both this and the NeurIPS paper are demonstration papers (They are empirically demonstrating an unintuitive phenomenon). Both papers are demonstrating that robust models learn features that correspond to salient parts of the input. Even though both papers are nice, either one is sufficient to demonstrate the phenomenon. For this to be a stand-alone paper, the authors would have to do more in my opinion. One option would be to explore and compare different forms of adversarial robustness as priors (The paper is called "Adversarial Robustness as a Prior for Learned Representations" and not "L2 Adversarial Robustness as a Prior for Learned Representations," after all). Another option would be to see if such representations are 'quantitatively' better in some settings (Such as for transfer learning). 
In its current form, I feel that the two papers are too similar to recommend acceptance.