Neural networks trained through stochastic gradient descent (SGD) have been around for more than 30 years, but they still escape our understanding. This paper takes an experimental approach, with a divide-and-conquer strategy in mind: we start by studying what happens in single neurons. While being the core building block of deep neural networks, the way they encode information about the inputs and how such encodings emerge is still unknown. We report experiments providing strong evidence that hidden neurons behave like binary classifiers during training and testing. During training, analysis of the gradients reveals that a neuron separates two categories of inputs, which for layers close enough to the output remain impressively constant across training. During testing, we show that the fuzzy, binary partition described above embeds the core information used by the network for its prediction. These observations bring to light some of the core internal mechanics of deep neural networks, and have the potential to guide the next theoretical and practical developments.
--------------------
Review updates:
Rating 6 -> 7
Confidence 2 -> 4
The rebuttal and update addressed a number of my concerns, cleared up confusing sections, and moved the paper materially closer to being publication-worthy, thus I’ve increased my score.
--------------------
I want to love this paper. The results seem like they may be very important. However, a few parts were poorly explained, which led to this reviewer being unable to follow some of the jumps from experimental results to their conclusions. I would like to be able to give this paper the higher score it may deserve, but some parts first need to be further explained.
Unfortunately, the largest single confusion I had is on the first, most basic set of gradient results of section 4.1. Without understanding this first result, it’s difficult to decide to what extent the rest of the paper’s results are to be believed.
Fig 1 shows “the histograms of the average sign of partial derivatives of the loss with respect to activations, as collected over training for a random neuron in five different layers.” Let’s consider the top-left subplot of Fig 1, showing a heavily bimodal distribution (modes near -1 and +1.). Is this plot made using data from a single neuron or from  multiple neurons? For now let’s assume it is for a single neuron, as the caption and text in 4.1 seem to suggest. If it is for a single neuron, then that neuron will have, for a single input example, a single scalar activation value and a single scalar gradient value. The sign of the gradient will either be +1 or -1. If we compute the sign for each input example and then AGGREGATE over all training examples seen by this neuron over the course of training (or a subset for computational reasons), this will give us a list of signs. Let’s collect these signs into a long list: [+1, +1, +1, -1, +1, +1, …]. Now what do we do with this list? As far as I can tell, we can either average it (giving, say, .85 if the list has far more +1 values than -1 values) OR we can show a histogram of the list, which would just be two bars at -1 and +1. But we can’t do both, indicating that some assumption above was incorrect. Which assumption in reading the text was incorrect?
Further in this direction, Section 4.1 claims “Zero partial derivatives are ignored to make the signal more clear.” Are these zero partial derivatives of the post-relu or pre-relu? The text (Sec 3) points to activations as being post-relu, but in this case zero-gradients should be a very small set (only occuring if all neurons on the next layer had either zero pre-relu gradients, which is common for individual neurons but, I would think, not for all at once). Or does this mean the pre-relu gradient is zero, e.g. the common case where the gradient is zeroed because the pre-activation was negative and the relu at that point has zero slope? In this case we would be excluding a large set (about half!) of the gradient values, and it didn’t seem from the context in the paper that this would be desirable.
It would be great if the above could be addressed. Below are some less important comments.
Sec 5.1: great results!
Fig 3: This figure studies “the first and last layers of each network”. Is the last layer really the last linear layer, the one followed by a softmax? In this case there is no relu and the 0 pre-activation is not meaningful (softmax is shift invariant). Or is the layer shown (e.g. “stage3layer2”) the penultimate layer? Minor: in this figure, it would be great if the plots could be labeled with which networks/datasets they are from.
Sec 5.2 states “neuron partitions the inputs in two distinct but overlapping categories of quasi equal size.” This experiment only shows that this is true in aggregate, not for specific neurons? I.e. the partition percent for each neuron could be sampled from U(45, 55) or from U(10, 90) and this experiment would not tell us which, correct? Perhaps this statement could be qualified.
Table 1: “52th percentile vs actual 53 percentile shown”. 
> Table 1: The more fuzzy, the higher the percentile rank of the threshold
This is true for the CIFAR net but the opposite is true for ResNet, right?