Neural ordinary differential equations (ODEs) have been attracting increasing attention in various research domains recently. There have been some works studying optimization issues and approximation capabilities of neural ODEs, but their robustness is still yet unclear. In this work, we fill this important gap by exploring robustness properties of neural ODEs both empirically and theoretically. We first present an empirical study on the robustness of the neural ODE-based networks (ODENets) by exposing them to inputs with various types of perturbations and subsequently investigating the changes of the corresponding outputs. In contrast to conventional convolutional neural networks (CNNs), we find that the ODENets are more robust against both random Gaussian perturbations and adversarial attack examples. We then provide an insightful understanding of this phenomenon by exploiting a certain desirable property of the flow of a continuous-time ODE, namely that integral curves are non-intersecting. Our work suggests that, due to their intrinsic robustness, it is promising to use neural ODEs as a basic block for building robust deep network models. To further enhance the robustness of vanilla neural ODEs, we propose the time-invariant steady neural ODE (TisODE), which regularizes the flow on perturbed data via the time-invariant property and the imposition of a steady-state constraint. We show that the TisODE method outperforms vanilla neural ODEs and also can work in conjunction with other state-of-the-art architectural methods to build more robust deep networks.
This paper studied the robustness of neural ODE-based networks (ODENets) to various types of perturbations on the input images. The authors observed that ODENets are more robust to both Gaussian perturbation and adversarial attacks, which the authors explained as non-intersecting of the integral curves for different initial points. Moreover, the authors proposed the time-invariant steady neural ODE (TisODE) to enhance the robustness of ODENets. I list my concerns below:
1. To show the ODENet is more robust, the authors should bound the gap between the integral curves for different inputs. Non-intersecting of the integral curves does not guarantee the robustness. 
2. The ODENet architecture showed in Figure~1 can be regarded as an augmented CNN. I think the identity map gives a good trade-off between robustness and generalization. To enhance robustness, one might design an expansion map, but this, in general, hurt the accuracy of the model.
3. Why do not perform experiments on the CIFAR10 benchmark dataset? I think it is very important to add these results.
4. To verify the robustness of ODENets and CNNs, the authors should also perform adversarial training besides training on the original and noisy images with Gaussian perturbation.
5. Theorem~1 is wrong. I suggest the authors check the conditions to make it valid. We can construct an ODE of the form (1) that blows up in finite-time, e.g., dx/dt = x^2.
6. Most importantly, the authors should match the number of function evaluations of neural ODE with the depth of the CNN, in addition to matching the number of parameters. Please perform such a comparison in rebuttal. (THIS IS THE MOST IMPORTANT RESULT I WANT TO SEE IN REBUTTAL)
7. The authors did not compare with existing work that tries to improve the robustness of neural nets from a differential equation viewpoint. The related works should be elaborated.
======================
I would like to point out a few related papers that lift the dimension of ODE to a transport equation and improve the neural nets' robustness from the lens of the transport equation's theory. Also, the author should compare their results with some reported results in 1, 3, 4.
1. Bao Wang, Binjie Yuan, Zuoqiang Shi, Stanley J. Osher. ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies, arXiv:1811.10745, NeurIPS, 2019
2. Bao Wang, Xiyang Luo, Zhen Li, Wei Zhu, Zuoqiang Shi, Stanley J. Osher. Deep Neural Nets with Interpolating Function as Output Activation, NeurIPS, 2018
3. Bao Wang, Alex T. Lin, Zuoqiang Shi, Wei Zhu, Penghang Yin, Andrea L. Bertozzi, Stanley J. Osher. Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization, arXiv:1809.08516, 2018
4. B. Wang, S. Osher. Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning, arXiv:1907.06800
=======================
Please address the previously mentioned concerns in rebuttal.