Improving Invariance and Equivariance Properties of Convolutional Neural Networks
Convolutional Neural Networks (CNNs) learn highly discriminative representations from data, but how robust and structured are these representations? How does the data shape the internal network representation? We shed light on these questions by empirically measuring the invariance and equivariance properties of a large number of CNNs trained with various types of input transformations. We find that CNNs learn invariance wrt all 9 tested transformation types and that invariance extends to transformations outside the training range. We also measure the distance between CNN representations and show that similar input transformations lead to more similar internal representations. Transforms can be grouped by the way they affect the learned representation. Additionally, we also propose a loss function that aims to improve CNN equivariance.
This paper is an extension of Lenc&Vedaldi15 paper, showing CNN representations at FC7 layer are to certain extent equivariant to various classes of transformations and that training with a certain group of transformation makes the representations more equivalent. Authors performed a large amount of experiments, training over 30 networks with different forms of jitter, which is quite impressive. However it is rather difficult to find a main message of this work. Yes, authors measured the properties on a different layer than the Lenc&Vedaldi15, however it is hard to find some novel insights other than the known fact that jitter helps to achieve invariance. The evaluation seems to be mostly correct, however the paper does not seem to be solving the task advertised in its title really well. Major issues are in the experiments with the representation distances: * The selection of only FC7 is a bit controversial - it is followed only by a single classification layer (FC8) to the common output - class likelyhoods. Because the FC8 is just a linear projections, what the equivalence map does is just to re-project the FC8 weights of the attached network to the weights of the original network. Probably performing similar experiments but on more layers may be more useful (as the networks are already trained). * The experiment with representation distance is missing what is the classification error on the testing dataset. This would answer whether the representations are actually compatible up to linear transformation at all... * It is not clear for the experiment with K-NN whether this is measured per each test set example? After training the equivalence map? More clear would be to show that networks trained on similar group of jitter transformations are more compatible on the target task. * The proposed method does not seem to improve equivariance consistently on all tasks. Especially with lambda_1 and lambda_2 having such small values, the loss is basically equal to simple data jitter as it just adds up the loss of the original and transformed image. Maybe the issue is in the selection of the FC7 layer? In general, this paper shows some interesting results on the FC7 equivariance, but it does not seem to be drawing many interesting new observations out of these experiments. Due to some issues with the equivalence experiments and the finetuning of equivariance, I would not recommend acceptance of this manuscript. However, refining the experiments on already trained networks and restructuring this manuscript into more investigative work may lead to interesting contribution to the field. There are also few minor issues: * It is not experimentally verified that the new criterion for equivariance mapping helps to gain better results. * The angles on page 1 and 5 are missing units (degrees?). * On page three, *In practice, it is difficult... *, it is not M_g which is maximised/minimised, but the loss over the M_g * Page 4, footnote 2 - if you are just halving the activations, it is hard to call it a dropout as this constant factor can be passed to the following/preceding weights * Is the network for RVL-CDIP the same architecture as Alexnet? * On page 7, Figure 3a+3b - in my opinion, turning the diagonal elements to white is really misleading, and probably even incorrect, as the distance between the same representations should be zero (which is also a way how to verify that the experiments are performed correctly).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Improving Invariance and Equivariance Properties of Convolutional Neural Networks
Convolutional Neural Networks (CNNs) learn highly discriminative representations from data, but how robust and structured are these representations? How does the data shape the internal network representation? We shed light on these questions by empirically measuring the invariance and equivariance properties of a large number of CNNs trained with various types of input transformations. We find that CNNs learn invariance wrt all 9 tested transformation types and that invariance extends to transformations outside the training range. We also measure the distance between CNN representations and show that similar input transformations lead to more similar internal representations. Transforms can be grouped by the way they affect the learned representation. Additionally, we also propose a loss function that aims to improve CNN equivariance.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper is an extension of Lenc&Vedaldi15 paper, showing CNN representations at FC7 layer are to certain extent equivariant to various classes of transformations and that training with a certain group of transformation makes the representations more equivalent. Authors performed a large amount of experiments, training over 30 networks with different forms of jitter, which is quite impressive. However it is rather difficult to find a main message of this work. Yes, authors measured the properties on a different layer than the Lenc&Vedaldi15, however it is hard to find some novel insights other than the known fact that jitter helps to achieve invariance. The evaluation seems to be mostly correct, however the paper does not seem to be solving the task advertised in its title really well. Major issues are in the experiments with the representation distances: * The selection of only FC7 is a bit controversial - it is followed only by a single classification layer (FC8) to the common output - class likelyhoods. Because the FC8 is just a linear projections, what the equivalence map does is just to re-project the FC8 weights of the attached network to the weights of the original network. Probably performing similar experiments but on more layers may be more useful (as the networks are already trained). * The experiment with representation distance is missing what is the classification error on the testing dataset. This would answer whether the representations are actually compatible up to linear transformation at all... * It is not clear for the experiment with K-NN whether this is measured per each test set example? After training the equivalence map? More clear would be to show that networks trained on similar group of jitter transformations are more compatible on the target task. * The proposed method does not seem to improve equivariance consistently on all tasks. Especially with lambda_1 and lambda_2 having such small values, the loss is basically equal to simple data jitter as it just adds up the loss of the original and transformed image. Maybe the issue is in the selection of the FC7 layer? In general, this paper shows some interesting results on the FC7 equivariance, but it does not seem to be drawing many interesting new observations out of these experiments. Due to some issues with the equivalence experiments and the finetuning of equivariance, I would not recommend acceptance of this manuscript. However, refining the experiments on already trained networks and restructuring this manuscript into more investigative work may lead to interesting contribution to the field. There are also few minor issues: * It is not experimentally verified that the new criterion for equivariance mapping helps to gain better results. * The angles on page 1 and 5 are missing units (degrees?). * On page three, *In practice, it is difficult... *, it is not M_g which is maximised/minimised, but the loss over the M_g * Page 4, footnote 2 - if you are just halving the activations, it is hard to call it a dropout as this constant factor can be passed to the following/preceding weights * Is the network for RVL-CDIP the same architecture as Alexnet? * On page 7, Figure 3a+3b - in my opinion, turning the diagonal elements to white is really misleading, and probably even incorrect, as the distance between the same representations should be zero (which is also a way how to verify that the experiments are performed correctly).
assistant
This paper is an extension of Lenc&Vedaldi15 paper, showing CNN representations at FC7 layer are to certain extent equivariant to various classes of transformations and that training with a certain group of transformation makes the representations more equivalent. Authors performed a large amount of experiments, training over 30 networks with different forms of jitter, which is quite impressive. However it is rather difficult to find a main message of this work. Yes, authors measured the properties on a different layer than the Lenc&Vedaldi15, however it is hard to find some novel insights other than the known fact that jitter helps to achieve invariance. The evaluation seems to be mostly correct, however the paper does not seem to be solving the task advertised in its title really well. Major issues are in the experiments with the representation distances: * The selection of only FC7 is a bit controversial - it is followed only by a single classification layer (FC8) to the common output - class likelyhoods. Because the FC8 is just a linear projections, what the equivalence map does is just to re-project the FC8 weights of the attached network to the weights of the original network. Probably performing similar experiments but on more layers may be more useful (as the networks are already trained). * The experiment with representation distance is missing what is the classification error on the testing dataset. This would answer whether the representations are actually compatible up to linear transformation at all... * It is not clear for the experiment with K-NN whether this is measured per each test set example? After training the equivalence map? More clear would be to show that networks trained on similar group of jitter transformations are more compatible on the target task. * The proposed method does not seem to improve equivariance consistently on all tasks. Especially with lambda_1 and lambda_2 having such small values, the loss is basically equal to simple data jitter as it just adds up the loss of the original and transformed image. Maybe the issue is in the selection of the FC7 layer? In general, this paper shows some interesting results on the FC7 equivariance, but it does not seem to be drawing many interesting new observations out of these experiments. Due to some issues with the equivalence experiments and the finetuning of equivariance, I would not recommend acceptance of this manuscript. However, refining the experiments on already trained networks and restructuring this manuscript into more investigative work may lead to interesting contribution to the field. There are also few minor issues: * It is not experimentally verified that the new criterion for equivariance mapping helps to gain better results. * The angles on page 1 and 5 are missing units (degrees?). * On page three, *In practice, it is difficult... *, it is not M_g which is maximised/minimised, but the loss over the M_g * Page 4, footnote 2 - if you are just halving the activations, it is hard to call it a dropout as this constant factor can be passed to the following/preceding weights * Is the network for RVL-CDIP the same architecture as Alexnet? * On page 7, Figure 3a+3b - in my opinion, turning the diagonal elements to white is really misleading, and probably even incorrect, as the distance between the same representations should be zero (which is also a way how to verify that the experiments are performed correctly).