Intriguing Properties of Learned Representations
A key feature of neural networks, particularly deep convolutional neural networks, is their ability to learn useful representations from data. The very last layer of a neural network is then simply a linear model trained on these learned representations. Despite their numerous applications in other tasks such as classification, retrieval, clustering etc., a.k.a. transfer learning, not much work has been published that investigates the structure of these representations or indeed whether structure can be imposed on them during the training process. In this paper, we study the effective dimensionality of the learned representations by models that have proved highly successful for image classification. We focus on ResNet-18, ResNet-50 and VGG-19 and observe that when trained on CIFAR10 or CIFAR100, the learned representations exhibit a fairly low rank structure. We propose a modification to the training procedure, which further encourages low rank structure on learned activations. Empirically, we show that this has implications for robustness to adversarial examples and compression.
 Synopsis: Overall, this paper was fairly well written and seems to have an original approach towards inducing low-rank structure on the space of activations in some intermediate layer in a computationally efficient way without changing the underlying model. This training modification does not seem to affect test performance and the low-rank embeddings that are learned seem useful at discriminative tasks. Adversarial robustness also appears improved. Pros: --While I am not familiar enough with the background literature on model compression in neural networks, I thought the augmented optimization problem used to induce low-rank structure on the space of activations was interesting and worthy of investigation. The authors appear to get great results in Table 1 & Table 2. Cons: --I cannot really gauge the significance of the result against other existing approaches towards low-dimensional representations because of my limited familiarity with the relevant literature. However, I didn’t feel quite convinced by the discussion in the paper that low-rank activations were superior to other kinds of low-rank approximations, for instance to the network weights (c.f. discussion in Appendix A). I think the discussion on this topic could be a bit improved. --With respect to the writing, I’m a bit uncertain as to the primary message of the paper. While it seems to introduce a new augmented training approach for generating compressed representations which potentially has practical utility, based on the paper title and scattered discussion it seems to suggest that the representations themselves are interesting, e.g. the idea of having low-rank activations while largely maintaining test performance. I didn’t fully understand the extent to which the results are intriguing or helpful in understanding neural networks. Could this be developed a bit more? Miscellaneous comments: --In Figure 2, the accuracy with respect to adversarial perturbations seems to drop more for VGG19 2-LR (pink curve) than the model VGG19 N-LR (brown), which seems counter to your point on robustness? --In Figure 4, why is the behavior of ResNet 2-LR (blue curve) similar to ResNet N-LR (red curve)? I would’ve expected any number of LR layers to increase the sensitivity in intermediate layers to adversarial input perturbations.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Intriguing Properties of Learned Representations
A key feature of neural networks, particularly deep convolutional neural networks, is their ability to learn useful representations from data. The very last layer of a neural network is then simply a linear model trained on these learned representations. Despite their numerous applications in other tasks such as classification, retrieval, clustering etc., a.k.a. transfer learning, not much work has been published that investigates the structure of these representations or indeed whether structure can be imposed on them during the training process. In this paper, we study the effective dimensionality of the learned representations by models that have proved highly successful for image classification. We focus on ResNet-18, ResNet-50 and VGG-19 and observe that when trained on CIFAR10 or CIFAR100, the learned representations exhibit a fairly low rank structure. We propose a modification to the training procedure, which further encourages low rank structure on learned activations. Empirically, we show that this has implications for robustness to adversarial examples and compression.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Synopsis: Overall, this paper was fairly well written and seems to have an original approach towards inducing low-rank structure on the space of activations in some intermediate layer in a computationally efficient way without changing the underlying model. This training modification does not seem to affect test performance and the low-rank embeddings that are learned seem useful at discriminative tasks. Adversarial robustness also appears improved. Pros: --While I am not familiar enough with the background literature on model compression in neural networks, I thought the augmented optimization problem used to induce low-rank structure on the space of activations was interesting and worthy of investigation. The authors appear to get great results in Table 1 & Table 2. Cons: --I cannot really gauge the significance of the result against other existing approaches towards low-dimensional representations because of my limited familiarity with the relevant literature. However, I didn’t feel quite convinced by the discussion in the paper that low-rank activations were superior to other kinds of low-rank approximations, for instance to the network weights (c.f. discussion in Appendix A). I think the discussion on this topic could be a bit improved. --With respect to the writing, I’m a bit uncertain as to the primary message of the paper. While it seems to introduce a new augmented training approach for generating compressed representations which potentially has practical utility, based on the paper title and scattered discussion it seems to suggest that the representations themselves are interesting, e.g. the idea of having low-rank activations while largely maintaining test performance. I didn’t fully understand the extent to which the results are intriguing or helpful in understanding neural networks. Could this be developed a bit more? Miscellaneous comments: --In Figure 2, the accuracy with respect to adversarial perturbations seems to drop more for VGG19 2-LR (pink curve) than the model VGG19 N-LR (brown), which seems counter to your point on robustness? --In Figure 4, why is the behavior of ResNet 2-LR (blue curve) similar to ResNet N-LR (red curve)? I would’ve expected any number of LR layers to increase the sensitivity in intermediate layers to adversarial input perturbations.
assistant
 Synopsis: Overall, this paper was fairly well written and seems to have an original approach towards inducing low-rank structure on the space of activations in some intermediate layer in a computationally efficient way without changing the underlying model. This training modification does not seem to affect test performance and the low-rank embeddings that are learned seem useful at discriminative tasks. Adversarial robustness also appears improved. Pros: --While I am not familiar enough with the background literature on model compression in neural networks, I thought the augmented optimization problem used to induce low-rank structure on the space of activations was interesting and worthy of investigation. The authors appear to get great results in Table 1 & Table 2. Cons: --I cannot really gauge the significance of the result against other existing approaches towards low-dimensional representations because of my limited familiarity with the relevant literature. However, I didn’t feel quite convinced by the discussion in the paper that low-rank activations were superior to other kinds of low-rank approximations, for instance to the network weights (c.f. discussion in Appendix A). I think the discussion on this topic could be a bit improved. --With respect to the writing, I’m a bit uncertain as to the primary message of the paper. While it seems to introduce a new augmented training approach for generating compressed representations which potentially has practical utility, based on the paper title and scattered discussion it seems to suggest that the representations themselves are interesting, e.g. the idea of having low-rank activations while largely maintaining test performance. I didn’t fully understand the extent to which the results are intriguing or helpful in understanding neural networks. Could this be developed a bit more? Miscellaneous comments: --In Figure 2, the accuracy with respect to adversarial perturbations seems to drop more for VGG19 2-LR (pink curve) than the model VGG19 N-LR (brown), which seems counter to your point on robustness? --In Figure 4, why is the behavior of ResNet 2-LR (blue curve) similar to ResNet N-LR (red curve)? I would’ve expected any number of LR layers to increase the sensitivity in intermediate layers to adversarial input perturbations.