A convolutional neural network for image classification can be constructed mathematically since it can be regarded as a multi-period dynamical system. In this paper, a novel approach is proposed to construct network models from the dynamical systems view. Since a pre-activation residual network can be deemed an approximation of a time-dependent dynamical system using the forward Euler method, higher order Runge-Kutta methods (RK methods) can be utilized to build network models in order to achieve higher accuracy. The model constructed in such a way is referred to as the Runge-Kutta Convolutional Neural Network (RKNet). RK methods also provide an interpretation of Dense Convolutional Networks (DenseNets) and Convolutional Neural Networks with Alternately Updated Clique (CliqueNets) from the dynamical systems view. The proposed methods are evaluated on benchmark datasets: CIFAR-10/100, SVHN and ImageNet. The experimental results are consistent with the theoretical properties of RK methods and support the dynamical systems interpretation. Moreover, the experimental results show that the RKNets are superior to the state-of-the-art network models on CIFAR-10 and on par on CIFAR-100, SVHN and ImageNet.
The paper takes the perspective that a ResNet can be seen as a discrete time dynamical system approximation, and then uses that insight to introduce a Runge Kutta style continuous dynamical system.  Results are achieved that outperform DenseNet and CliqueNet
Quality
- The connection between deep nets and dynamical systems is an important one, as a small but growing literature demonstrates.
- The quality of this paper is difficult to extract, given rather long and unclear definitions of various key choices, at key points opting instead for a speculative connection to neurosciences (see clarity below).
- The empirical results could make this paper compelling, but the comparisons are rather shallow, in that they compare on only a few datasets and don't compare extensively to many of the state of the art methods.   For example, in Table 2 DenseNet outperforms the IRKNets, but this fact is not highlighted.  Overall this and other key omissions create the concern that the results are not particularly strong.
- One key reference that is missing is the careful work Chen et al https://arxiv.org/abs/1806.07366.  This work provides an excellent basis for how to present these ideas rigorously.
Clarity
- The connections to the brain -- such as "visual cortex", "ventral stream", etc -- are not rigorous and are unrelated to the paper at hand; really all connections to brain should be removed.  
- Figures 2 and 3 are the central architectural choices, but the writing around them does not clarify why all the choices have been made, and what the implications are.
Originality
- This is one of a small but growing number of papers connecting deep nets to ODE solvers.  Chen et al should be cited (Chen et al https://arxiv.org/abs/1806.07366).  
- There appears to be sufficient originality.
Significance
- hard to determine with unclear exposition and rather incomplete empirical results.