RESIDUAL NETWORKS CLASSIFY INPUTS BASED ON THEIR NEURAL TRANSIENT DYNAMICS
In this study, we analyze the input-output behavior of residual networks from a dynamical system point of view by disentangling the residual dynamics from the output activities before the classification stage. For a network with simple skip connections between every successive layer, and for logistic activation function, and shared weights between layers, we show analytically that there is a cooperation and competition dynamics between residuals corresponding to each input dimension. Interpreting these kind of networks as nonlinear filters, the steady state value of the residuals in the case of attractor networks are indicative of the common features between different input dimensions that the network has observed during training, and has encoded in those components. In cases where residuals do not converge to an attractor state, their internal dynamics are separable for each input class, and the network can reliably approximate the output. We bring analytical and empirical evidence that residual networks classify inputs based on the integration of the transient dynamics of the residuals, and will show how the network responds to input perturbations. We compare the network dynamics for a ResNet and a Multi-Layer Perceptron and show that the internal dynamics, and the noise evolution are fundamentally different in these networks, and ResNets are more robust to noisy inputs. Based on these findings, we also develop a new method to adjust the depth for residual networks during training. As it turns out, after pruning the depth of a ResNet using this algorithm,the network is still capable of classifying inputs with a high accuracy.
 This paper presents the following main insight (quoting the authors): Resnets classify input patterns based on the sum of the transient dynamics of the residuals in different layers. The authors formulate this insight mathematically, and experimentally demonstrate its validity on a toy binary classification problem. They also show that the behaviour of a Residual network trained on MNIST is in line with the claims. Finally, a method to adapt the depth of a Residual network during training is proposed and applied to the toy classification problem. The paper is generally of good quality and easy to understand. My only complaint: the introduction (including related work) is too long and I think it will be unclear to a general reader before reading section 2, where the terms used in the paper are explained and clarified. I think it will be an improvement to leave out detailed discussion of related work for a separate section, and focus on clarifying what the paper is about. Overall, while the paper is related to an interesting and potentially fruitful perspective of neural networks (as dynamical systems), in my view the contributions are not significant at this stage. That the sum of transients determines network outputs is almost by design, and can be shown without a dynamical systems perspective. Using the paper’s notation, one can sum over the equations for all the layers to obtain this. x(1) = x(0) + y(1) … x(T) = x(T-1) + y(T) ———————————————————————— x(T) = x(0) + sum(y(t)) Since the classification is performed using x(T), it is clear that the sum of transients is what causes the change in representation and that y(T) can be the same or not for different class inputs. Based on my understanding, I don’t find the findings to be significantly novel or surprising. In particular, I don’t see any concrete benefits of employing a dynamical systems perspective here. Nevertheless, the experimental analysis is interesting, and I’m hopeful that this direction will lead to more insights in the future. The final contribution is a new method to learn the depth of Residual networks during training, but this is insufficiently explored (only tested on the toy dataset) so its practical significance can not be evaluated at this stage. Minor notes: - Note that many citations are textual when they should be parenthetical. - The reference “No & Liao (2016)” has incorrect author name.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
RESIDUAL NETWORKS CLASSIFY INPUTS BASED ON THEIR NEURAL TRANSIENT DYNAMICS
In this study, we analyze the input-output behavior of residual networks from a dynamical system point of view by disentangling the residual dynamics from the output activities before the classification stage. For a network with simple skip connections between every successive layer, and for logistic activation function, and shared weights between layers, we show analytically that there is a cooperation and competition dynamics between residuals corresponding to each input dimension. Interpreting these kind of networks as nonlinear filters, the steady state value of the residuals in the case of attractor networks are indicative of the common features between different input dimensions that the network has observed during training, and has encoded in those components. In cases where residuals do not converge to an attractor state, their internal dynamics are separable for each input class, and the network can reliably approximate the output. We bring analytical and empirical evidence that residual networks classify inputs based on the integration of the transient dynamics of the residuals, and will show how the network responds to input perturbations. We compare the network dynamics for a ResNet and a Multi-Layer Perceptron and show that the internal dynamics, and the noise evolution are fundamentally different in these networks, and ResNets are more robust to noisy inputs. Based on these findings, we also develop a new method to adjust the depth for residual networks during training. As it turns out, after pruning the depth of a ResNet using this algorithm,the network is still capable of classifying inputs with a high accuracy.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper presents the following main insight (quoting the authors): Resnets classify input patterns based on the sum of the transient dynamics of the residuals in different layers. The authors formulate this insight mathematically, and experimentally demonstrate its validity on a toy binary classification problem. They also show that the behaviour of a Residual network trained on MNIST is in line with the claims. Finally, a method to adapt the depth of a Residual network during training is proposed and applied to the toy classification problem. The paper is generally of good quality and easy to understand. My only complaint: the introduction (including related work) is too long and I think it will be unclear to a general reader before reading section 2, where the terms used in the paper are explained and clarified. I think it will be an improvement to leave out detailed discussion of related work for a separate section, and focus on clarifying what the paper is about. Overall, while the paper is related to an interesting and potentially fruitful perspective of neural networks (as dynamical systems), in my view the contributions are not significant at this stage. That the sum of transients determines network outputs is almost by design, and can be shown without a dynamical systems perspective. Using the paper’s notation, one can sum over the equations for all the layers to obtain this. x(1) = x(0) + y(1) … x(T) = x(T-1) + y(T) ———————————————————————— x(T) = x(0) + sum(y(t)) Since the classification is performed using x(T), it is clear that the sum of transients is what causes the change in representation and that y(T) can be the same or not for different class inputs. Based on my understanding, I don’t find the findings to be significantly novel or surprising. In particular, I don’t see any concrete benefits of employing a dynamical systems perspective here. Nevertheless, the experimental analysis is interesting, and I’m hopeful that this direction will lead to more insights in the future. The final contribution is a new method to learn the depth of Residual networks during training, but this is insufficiently explored (only tested on the toy dataset) so its practical significance can not be evaluated at this stage. Minor notes: - Note that many citations are textual when they should be parenthetical. - The reference “No & Liao (2016)” has incorrect author name.
assistant
 This paper presents the following main insight (quoting the authors): Resnets classify input patterns based on the sum of the transient dynamics of the residuals in different layers. The authors formulate this insight mathematically, and experimentally demonstrate its validity on a toy binary classification problem. They also show that the behaviour of a Residual network trained on MNIST is in line with the claims. Finally, a method to adapt the depth of a Residual network during training is proposed and applied to the toy classification problem. The paper is generally of good quality and easy to understand. My only complaint: the introduction (including related work) is too long and I think it will be unclear to a general reader before reading section 2, where the terms used in the paper are explained and clarified. I think it will be an improvement to leave out detailed discussion of related work for a separate section, and focus on clarifying what the paper is about. Overall, while the paper is related to an interesting and potentially fruitful perspective of neural networks (as dynamical systems), in my view the contributions are not significant at this stage. That the sum of transients determines network outputs is almost by design, and can be shown without a dynamical systems perspective. Using the paper’s notation, one can sum over the equations for all the layers to obtain this. x(1) = x(0) + y(1) … x(T) = x(T-1) + y(T) ———————————————————————— x(T) = x(0) + sum(y(t)) Since the classification is performed using x(T), it is clear that the sum of transients is what causes the change in representation and that y(T) can be the same or not for different class inputs. Based on my understanding, I don’t find the findings to be significantly novel or surprising. In particular, I don’t see any concrete benefits of employing a dynamical systems perspective here. Nevertheless, the experimental analysis is interesting, and I’m hopeful that this direction will lead to more insights in the future. The final contribution is a new method to learn the depth of Residual networks during training, but this is insufficiently explored (only tested on the toy dataset) so its practical significance can not be evaluated at this stage. Minor notes: - Note that many citations are textual when they should be parenthetical. - The reference “No & Liao (2016)” has incorrect author name.