Successfully training deep neural networks often requires either batch normalization, appropriate weight initialization, both of which come with their own challenges. We propose an alternative, geometrically motivated method for training. Using elementary results from linear programming, we introduce Farkas layers: a method that ensures at least one neuron is active at a given layer. Focusing on residual networks with ReLU activation, we empirically demonstrate a significant improvement in training capacity in the absence of batch normalization or methods of initialization across a broad range of network sizes on benchmark datasets.
The paper introduces a new type of layer, Farkas layers, that are designed to ameliorate the "dying ReLU problem". The idea of repurposing Farkas' lemma from linear programming to a component in NNs is very nice. However, the paper doesn't have convincing baselines and doesn't dig deep enough into what the Farkas layer is actually doing.
Comments:
1. Start of section 2.2: “Previous work on the dying ReLU problem, or vanishing gradient problem, in the case of using sigmoid-like activation functions, has heavily revolved around novel weight initializations”. Dying ReLUs and vanishing gradients are different problems. In particular, it doesn’t make sense to talk about the dying ReLU problem for sigmoid-like activation functions.
2. Batchnorm (BN) consists in two operations: shifting and scaling. Both are relevant to vanishing gradients. However, only shifting (that is, subtracting the mean and resetting to a learned value) is relevant to dying ReLUs because rescaling the input to a ReLU by a positive number doesn’t affect whether the ReLU is subsequently ON or OFF.
3. Given point #2, a natural baseline to compare the Farkas layer against is a “pared-down BN”, which shifts but does not rescale.
4. Similarly, when combining the Farkas operation with BN, it might be worth considering keeping BN’s rescaling but dropping the shift -- since Farkas is analogous to the shift operation in BN.
5. I claimed above that Farkas is analogous to the shift in BN, but haven’t thought about it deeply. Do you agree? Any comments on how they differ and why?
6. Our contributions, p2: “We empirically show an approximate 20% improvement on the first epoch over only using batch normalization.” I’m not sure what to make of this; improvements on the first epoch are only useful if they lead to overall improvements. 
7. Figure 4 of “Shattered gradients”, https://arxiv.org/abs/1702.08591 looks at ReLU activations by layer, both with and without BN. It’s worth doing a similar analysis for Farkas layers. Concretely: how do Farkas layers change the pattern and frequency of activation, both at activation and during training?
8. Guaranteeing the activity of a single neuron per layer seems very weak. What is the empirical effect of Farkas on the number of live neurons? Is it really just making sure one ReLU is on, or does it do better? Is it possible to ensure more neurons are ON in each layer? The “shattered” paper above suggests BN sets close to half ReLUs as ON at initialization, and approximately controls how often ReLUs are ON or OFF during training via the shift. 
9. As a broader point, the paper proposes an algorithm based on a hypothesis: that having (at least one) ReLU on helps training. It’s worth digging into the hypothesis a bit rather than just plotting training curves. 
10. I would expect ResNets have much *less* of a problem with dying ReLUs than standard NNs because of the skip-connections. One would therefore expect Farkas layers to help more with standard NNs than ResNets. However, the reported results are for ResNets. What happens when there are no skip-connections?