Improving Invariance and Equivariance Properties of Convolutional Neural Networks
Convolutional Neural Networks (CNNs) learn highly discriminative representations from data, but how robust and structured are these representations? How does the data shape the internal network representation? We shed light on these questions by empirically measuring the invariance and equivariance properties of a large number of CNNs trained with various types of input transformations. We find that CNNs learn invariance wrt all 9 tested transformation types and that invariance extends to transformations outside the training range. We also measure the distance between CNN representations and show that similar input transformations lead to more similar internal representations. Transforms can be grouped by the way they affect the learned representation. Additionally, we also propose a loss function that aims to improve CNN equivariance.
This work presents an empirical study of the influence of different types of data augmentation on the performance of CNNs. It also proposes to incorporate additional loss functions to encourage approximate invariance or equivariance, and shows there are some benefits. The paper reads well and the objectives are clear. The study of invariances in CNNs is a very important topic, and advances in this area are greatly appreciated. The paper splits itself in two very different parts -- the empirical study of equivariances in existing CNNs, and the proposal of equivariance objectives. However, taken separately each of these two parts could be better executed. On the empirical study, its breath is relatively limited, and it*s hard to draw any far-reaching conclusions from it: - Only one network is studied; at least one other architecture would have made for better generalization. - Only one layer (fc7) is studied; this presents issues as the top layer is the most invariant. At least one convolutional layer (possibly more) should have been considered. - The reliance on the scanned text dataset does not help; however the ImageNet results are definitely very encouraging. It is nice to see how performance degrades with the degree of transformations, and the authors do interpret the results, but it would be better to see more analysis. There is only a limited set of conclusions that can be drawn from evaluating networks with jittered data. If the authors could propose some other interesting ways to assess the invariance and equivariance, they would potentially draw more insightful conclusions from it. On the proposed loss function, only a very quick treatment of it is given (Section 4, half a page). It does not differ too much from known invariance/equivariance objectives studied in the literature previously, e.g. Decoste and Scholkopf, *Training Invariant Support Vector Machines*, Machine Learning, 2002. I*m not sure that dividing the paper into these two different contributions is the best approach; they both feel a bit incomplete, and a full treatment of only one of them would make for an overall better paper.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Improving Invariance and Equivariance Properties of Convolutional Neural Networks
Convolutional Neural Networks (CNNs) learn highly discriminative representations from data, but how robust and structured are these representations? How does the data shape the internal network representation? We shed light on these questions by empirically measuring the invariance and equivariance properties of a large number of CNNs trained with various types of input transformations. We find that CNNs learn invariance wrt all 9 tested transformation types and that invariance extends to transformations outside the training range. We also measure the distance between CNN representations and show that similar input transformations lead to more similar internal representations. Transforms can be grouped by the way they affect the learned representation. Additionally, we also propose a loss function that aims to improve CNN equivariance.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This work presents an empirical study of the influence of different types of data augmentation on the performance of CNNs. It also proposes to incorporate additional loss functions to encourage approximate invariance or equivariance, and shows there are some benefits. The paper reads well and the objectives are clear. The study of invariances in CNNs is a very important topic, and advances in this area are greatly appreciated. The paper splits itself in two very different parts -- the empirical study of equivariances in existing CNNs, and the proposal of equivariance objectives. However, taken separately each of these two parts could be better executed. On the empirical study, its breath is relatively limited, and it*s hard to draw any far-reaching conclusions from it: - Only one network is studied; at least one other architecture would have made for better generalization. - Only one layer (fc7) is studied; this presents issues as the top layer is the most invariant. At least one convolutional layer (possibly more) should have been considered. - The reliance on the scanned text dataset does not help; however the ImageNet results are definitely very encouraging. It is nice to see how performance degrades with the degree of transformations, and the authors do interpret the results, but it would be better to see more analysis. There is only a limited set of conclusions that can be drawn from evaluating networks with jittered data. If the authors could propose some other interesting ways to assess the invariance and equivariance, they would potentially draw more insightful conclusions from it. On the proposed loss function, only a very quick treatment of it is given (Section 4, half a page). It does not differ too much from known invariance/equivariance objectives studied in the literature previously, e.g. Decoste and Scholkopf, *Training Invariant Support Vector Machines*, Machine Learning, 2002. I*m not sure that dividing the paper into these two different contributions is the best approach; they both feel a bit incomplete, and a full treatment of only one of them would make for an overall better paper.
assistant
This work presents an empirical study of the influence of different types of data augmentation on the performance of CNNs. It also proposes to incorporate additional loss functions to encourage approximate invariance or equivariance, and shows there are some benefits. The paper reads well and the objectives are clear. The study of invariances in CNNs is a very important topic, and advances in this area are greatly appreciated. The paper splits itself in two very different parts -- the empirical study of equivariances in existing CNNs, and the proposal of equivariance objectives. However, taken separately each of these two parts could be better executed. On the empirical study, its breath is relatively limited, and it*s hard to draw any far-reaching conclusions from it: - Only one network is studied; at least one other architecture would have made for better generalization. - Only one layer (fc7) is studied; this presents issues as the top layer is the most invariant. At least one convolutional layer (possibly more) should have been considered. - The reliance on the scanned text dataset does not help; however the ImageNet results are definitely very encouraging. It is nice to see how performance degrades with the degree of transformations, and the authors do interpret the results, but it would be better to see more analysis. There is only a limited set of conclusions that can be drawn from evaluating networks with jittered data. If the authors could propose some other interesting ways to assess the invariance and equivariance, they would potentially draw more insightful conclusions from it. On the proposed loss function, only a very quick treatment of it is given (Section 4, half a page). It does not differ too much from known invariance/equivariance objectives studied in the literature previously, e.g. Decoste and Scholkopf, *Training Invariant Support Vector Machines*, Machine Learning, 2002. I*m not sure that dividing the paper into these two different contributions is the best approach; they both feel a bit incomplete, and a full treatment of only one of them would make for an overall better paper.