Training Very Deep Networks
Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.
I like the paper, the idea is well described and the experiments are convincing to a certain degree. The best thing in my opinion is that the authors tried to analyze the learned networks with respect to the pattern of gate outputs.(1) The gate unit T is modeled with a standard sigmoid activation. Therefore, the response value is never exactly 0 or 1 (this was also stated by the authors) and the gradients in eq. 5 are not correct. The authors should explain in the paper how backpropagation is exactly performed in these networks.(2) The effect of the initialization is always tricky to analyze, however, the fact that the authors propose to use constant negative biases depending on the number of levels is also not satisfying. I would like to see a plot of the performance with respect to the initial value of the bias. The initial bias acts as a kind of regularizer in this case with a high negative bias supporting networks with "longer highways" (I like this comparison by the way). (3) "report by recent studies in similar experimental settings" This sentence is unsatisfying. Either the methods use the same experimental setting, which would allow for comparison, or not, which would forbid a direct comparison. The authors need to clarify the differences of the experimental setups in the paper. (4) Giving "max" accuracies in tables is rather uncommon and also not reasonable from a statistical point of view. The authors should therefore always give mean accuracy and standard deviation for every method (and when available). (5) It would be interesting to study the proposed approach from a learning theory perspective.