Tensor Switching Networks
We present a novel neural network algorithm, the Tensor Switching (TS) network, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to tensor-valued hidden units. The TS network copies its entire input vector to different locations in an expanded representation, with the location determined by its hidden unit activity. In this way, even a simple linear readout from the TS representation can implement a highly expressive deep-network-like function. The TS network hence avoids the vanishing gradient problem by construction, at the cost of larger representation size. We develop several methods to train the TS network, including equivalent kernels for infinitely wide and deep TS networks, a one-pass linear learning algorithm, and two backpropagation-inspired representation learning algorithms. Our experimental results demonstrate that the TS network is indeed more expressive and consistently learns faster than standard ReLU networks.
The paper proposes a new network architecture that, like ReLU networks, is conditionally linear.  The novelty in the proposed architecture is that the weights that determine which units are active (called activation weights in the paper) are decoupled from the weights that map the input to the output (called analysis weights in the paper).  In the proposed architecture, active units copy their full input to the output, while inactive units output zeros.  As a consequence of this design, the output of each layer is a tensor, and each successive layer in the network increases the order of the tensor by one.  The analysis weights are applied only to the final output.  Three different algorithms for training these tensor switching networks are described in the paper.  The first derives a kernel function that is equivalent to one or more tensor switching layers that are infinitely wide and have activation weights drawn from a standard normal distribution.  Such kernels may be used with small data sets to learn SVMs, and comparisons between the tensor switching kernel and the arccosine kernel (which is the equivalent kernel for deep ReLU networks) provide insights into the differences between tensor switching and ReLU networks.  The second training algorithm draws the activation weights randomly from a given distribution and then uses regularized ridge regression to learn the analysis weights using only a single pass over the training data.  The third training algorithm introduces introduces a set of auxiliary variables such that the derivatives of the loss with respect to the auxiliary variables serve as proxies for the derivatives of the loss with respect to the unit activation weights, which are zero or undefined everywhere.  Experiments on variations of MNIST show that the combination of tensor switching networks and the second training algorithm provide the best performance when learning is done in a single pass over the training data. 
This is a creative piece of work, and the provision of three different learning algorithms that work and help provide insights into the proposed tensor switching architecture is impressive.  I have only a few minor comments that, if addressed, would improve the paper.  Figures 2a, b, and c would be clearer if the x axes were labeled in terms of fractions of pi.  Page 7, lines 242-243: "In terms of running time, the SS network enjoys a roughly 8× improvement in wall-clock training speed over the SS network for the 1 layer MLP." Should this be "...the RR network enjoys..."?  Page 7, lines 247-248: "the TS net architecture, [which] attains much lower asymptotic training error than the SS net but similar test errors" -- basically, TS networks overfit more than the SS networks. But this isn't a surprise, is it? The TS networks have far more parameters than the SS networks with the same architecture. One way of looking at this is that the TS networks trade off a large number of parameters for ease and speed of training.  How were the sizes of the MLPs in the experiments selected?  The one-pass training algorithm with ridge regression is somewhat reminiscent of one-pass training using randomized decision trees. The theory developed for randomized decision trees might provide a useful path for theoretical analysis of TS networks with random activation weights.