Workshop track -ICLR 2018 GRADIENTS EXPLODE -DEEP NETWORKS ARE SHALLOW -RESNET EXPLAINED
Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exploding gradients occur and highlight the collapsing domain problem, which can arise in architectures that avoid exploding gradients. ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks, which we show is a consequence of a surprising mathematical property. By noticing that any neural network is a residual network, we devise the residual trick, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.
Summary of paper - The paper introduces the Gradient Scale Coefficient and uses it to demonstrate issues with the current understanding of where and why exploding gradients occur. 
Review - The paper attempts to contribute to the discussion about the exploding gradient problem by both introducing a metric for discussing this issue and by showing that current understanding of the exploding gradient problem may be incorrect. It is admirable that the authors are seeking to add to the understanding about theory of neural nets instead of contributing a new architecture with better error rates but without understanding why said error rates are lower. While the authors list 7 contributions, the current version of the text is a challenge to read and makes it challenging to distill an overarching theme or narrative to these contributions. 
The authors do mention experiments on page 8, but confess that some of the results are somewhat underwhelming. Unfortunately, all tables with the experimental results are left to the appendix. As this is a mostly theoretical paper, pushing experimental results to the appendix does make sense, but the repeated references to these tables suggest that these experimental results are crucial for the authors’ overall points.
While the authors do attempt to accomplish a lot in these nearly 16 pages of text, the authors' main points and overall narrative gets lost due to the writing that is a bit jumbled at times and that relies heavily on the supplement. There are several places where it is not immediately clear why a certain block of text is included (i.e. the proof outlines on pages 8 and 10). At other points the authors default to an chronological narrative that can be useful at times (i.e. page 9), but here seems to distract from their overall narrative. 
This paper has a lot of content, but not all of it appears to be relevant to the authors’ central points. Furthermore, the paper is nearly double the recommended page length and has a nearly 30 page supplement. My biggest recommendations for this paper are for the authors to 1) articulate one theme and then 2) look at each part (whether that be section, paragraph, or sentence) and ask what does that part contribute to that theme. 
Pros - 
* This paper attempts to add the understanding of neural nets instead of only contributing better error rates on benchmark datasets. 
* At several points, the authors seek to make the work accessible by offering lay explanations for their more technical points. 
* The practical suggestions on page 16 are a true highlight and could provide an outline for possible revisions. 
Cons - 
* The main narrative is lost in the text, leaving a reader unsure of the authors main points and contributions as they read. For example, the authors’ first contribution is hidden among the text presentation of section 2. 
* The paper relies heavily on the supplement to make their central points. 
* It is nearly double the recommended page length with a nearly 30 page supplement
Minor issues - 
* Use one style for introducing and defining terms either use italics or single quotes. The latter is not recommended since the authors use double quotes in the abstract to express that the exploding gradient problem is not solved. 
* The citation style of Authors (YEAR) at times leads to awkward sentence parsing. 
* Given that many figures have several subfigures, the authors should consider using a package that will denote subfigures with letters. 
* The block quotes in the introduction may be quite important for points later in the paper, but summarizing the points of these quotes may be a better use of space. The authors more successfully did this in paragraph 2 of the introduction. 
* All long descriptions of the appendix should be carefully revisited and possibly removed due to page length considerations. 
* In the text, figure 4 (which is in the supplement) is referenced before figure 3 (which is in the text).
=-=-=-= Response to the authors
During the initial reviewing period, I was unable to distill the significance of the authors’ contributions from the current literature in large part due to the nature of the writing style. After reading the authors responses and consulting the differences between the versions of the paper, my review remains the same. It should be noted that all three reviewers pointed out the length of the paper as a weakness of the paper, and that in the most recent draft, the authors made the main text of the paper longer. 
Consulting the differences between the paper revisions, I was initially intrigued with the volume of differences that shown in the summary bar. Upon closer inspection, I read a much stronger introduction and appreciated the summaries at the ends of sections 4.4 and 6. However, I did notice that the majority of these changes were superficial re-orderings of the original text. Given the limited substantive changes to the main text, I did not deeply re-read the text of the paper beyond the introduction.