A key feature of neural networks, particularly deep convolutional neural networks, is their ability to "learn" useful representations from data. The very last layer of a neural network is then simply a linear model trained on these "learned" representations. Despite their numerous applications in other tasks such as classification, retrieval, clustering etc., a.k.a. transfer learning, not much work has been published that investigates the structure of these representations or indeed whether structure can be imposed on them during the training process. In this paper, we study the effective dimensionality of the learned representations by models that have proved highly successful for image classification. We focus on ResNet-18, ResNet-50 and VGG-19 and observe that when trained on CIFAR10 or CIFAR100, the learned representations exhibit a fairly low rank structure. We propose a modification to the training procedure, which further encourages low rank structure on learned activations. Empirically, we show that this has implications for robustness to adversarial examples and compression.
This paper introduces a new way to have more compressed (lower rank) representation of the data in a supervised fashion. The authors motivates their work by saying that such representation are more useful for transfer learning and are more robust to adversarial examples! In order to achieve this, the authors introduce a virtual LR layer and utilize Nystrom technique to make the process more efficient.  The idea introduced in this paper is interesting but the paper is poorly written and organized which makes its through evaluation difficult. Below, I provide more detailed comments.
I don't understand at what frequency the low-rank optimization as the  subproblem to Equation (OPT) is being done. Given that the DNN training  is being done using batched of examples, where do you put the low-rank  optimization, in the end of each epoch? It seems it is in the end of each epoch but it should be clearly stated.
I don't understand motivation for L_N(A). The authors justify it by  saying that a trivial solution for the optimization problem would be setting A+b=0 and they introduce this term to avoid it. However, this is not correct. Note that there are n examples in A and we can only make one of them  zero at a time. (Note that talking about A+b is not also accurate  because they are not of the same size).
In order to use Nystrom method for low-rank approximation, W needs to be  a symmetric postive semi-definite matrix. I am not sure if the heuristic  procedure introduced in Page 4 is well-justifed. and whether we are  still optimizing the objective function introduced in Page 3. Do we  still have a stable training?
What is ResNet N-LR in experiment? The authors introduced ResNet 1-LR and ResNet 2-LR but not ResNet N-LR! I found the description N-LR later but the naming is rather confusing. I would use LR instead of N-LR because it seems it has N LR layer. I would also explain this next to other methods. 
Not sure if I understand Bottle-LR. The description in the text is not clear and I don’t understand the motivation for this baseline! Again, this should be described next to other methods.
The authors do not mention what is their setting for r in the experiments (Table 1).
Page 5, Paragraph after Table 1: First CIFAR-100 should be CIFAR-10.
One way the authors defend their framework is to have a representation that can be used in transfer learning. Nonetheless, the results in Table 1.b shows that their framework is not doing good for transfer learning.
Not sure if I understand Figure 1. How do you change the number of singular values? I understand this is a hyper parameter for your framework but I am confused how it is being set for N-LR method. Similarly, I don’t understand Table 2 and how you change the embedding dimension. 
Unlike the claim made by the authors, it seems that VGG19 N-LR does better compared to VGG19 2-LR in Figure 2.