This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks. We generalise existing results, providing an alternative, simpler method for lower bounding expected trajectory growth through random networks, for a more general class of weights distributions, including sparsely connected networks. We illustrate this approach by deriving bounds for sparse-Gaussian, sparse-uniform, and sparse-discrete-valued random nets. We prove that trajectory growth can remain exponential in depth with these new distributions, including their sparse variants, with the sparsity parameter appearing in the base of the exponent.
In this paper, how the length of the trajectory in the input space is amplified by a ReLU neural network is analyzed. Specifically, the paper studied the case when the weights and biases are sparse random matrices. Some theoretical lower bounds are derived and are also empirically checked. 
Though the results are interesting, I am slightly lean to the rejection side. The main reason is that the motivation is not strong enough and it makes the entire work somehow incremental.
As described in the paper, the analysis of the trajectory length of NN has been initiated by Raghu et al. (2017). Although Raghu et al. considered the case of densely connected NNs, this work extended the notion to the sparsely connected NNs. I feel the extension is reasonable but it is not a big "jump" from the original work, and the originality is less significant. A good point of the derived results is that they are simple and easy to understand the dependency of the variance of the weights and the sparsity level. However, I am not so excited about these results because I cannot find a practical value from them. For example, (the current form of) Corollary 1 does not tell us how to control the sparsity level \alpha to maintain some accuracy-sparsity tradeoff. 
From the technical side, it is nice the proof is written in line by line. However, it looks everything is done in a normal manner and nothing special happens. Also, the condition of Theorem 2 is unclear. What does the "restriction" of \hat{w}_i mean? Does the condition E[] >= M ||u|| hold for any u and M? 
Finally, I have a simple question about the relationship between the scale of the output and the trajectory length. Let W be a weight and x be an input. When the smallest singular value of W is m, x is amplified at least by m, i.e., ||Wx|| >= m||x|| for any x. This means that (a part of) the trajectory growth is explained by the smallest singular value of the sparse random matrix W. Can you clarify the difference?