The gap between the empirical success of deep learning and the lack of strong theoretical guarantees calls for studying simpler models. By observing that a ReLU neuron is a product of a linear function with a gate (the latter determines whether the neuron is active or not), where both share a jointly trained weight vector, we propose to decouple the two. We introduce GaLU networks -networks in which each neuron is a product of a Linear Unit, defined by a weight vector which is being trained, with a Gate, defined by a different weight vector which is not being trained. Generally speaking, given a base model and a simpler version of it, the two parameters that determine the quality of the simpler version are whether its practical performance is close enough to the base model and whether it is easier to analyze it theoretically. We show that GaLU networks perform similarly to ReLU networks on standard datasets and we initiate a study of their theoretical properties, demonstrating that they are indeed easier to analyze. We believe that further research of GaLU networks may be fruitful for the development of a theory of deep learning.
The paper introduces a GaLU activation function, which is the product of a random gate function and a learnable linear function. The authors argue that empirically, neural networks with the GaLU activation is as effective as that with the ReLU activation, but theoretically, the GaLU activation is easier to understand because of the separation of the non-linearity and the learnable parameters. The the paper analyzes neural networks with one GaLU layer. Essentially, the network is a random transformation followed by a linear projection. This property enables analysis that are well known for the linear models.
Although the definition of GaLU is new, the idea of combining a non-linear projection with a linear transformation is an old one. [1] shows that many kernel SVM models can be written in this form. [2] [3] show that neural networks with various of activation functions can be relaxed to this form. However, these methods have never achieved performance that is as good as the state-of-the-art CNN models in challenging datasets (ImageNet or even CIFAR-10).
In section 3, the accuracies on MNIST (98%) and MNIST-fashion (88%) are quite low. They are not even as good as a classical kernel SVM, though the non-linear projection version of the kernel SVM has been well-studied in theory.
In section 4, the analyses are mostly standard for linear models and convex optimization. To the best of our knowledge, it doesn't introduce new insight on the understanding of non-convex optimization.
Overall, I think the paper and its theoretical analysis is built on an unsolid claim that the GaLU activation is a good replacement for traditional non-linear activation functions. The empirical study doesn't seem to support this claim. I cannot recommend accepting the paper.
[1] Random Features for Large-Scale Kernel Machines 
[2] Learning Kernel-Based Halfspaces with the Zero-One Loss
[3] Convexified Convolutional Neural Networks