G-SGD: OPTIMIZING RELU NEURAL NETWORKS IN ITS POSITIVELY SCALE-INVARIANT SPACE
It is well known that neural networks with rectified linear units (ReLU) activation functions are positively scale-invariant. Conventional algorithms like stochastic gradient descent optimize the neural networks in the vector space of weights, which is, however, not positively scale-invariant. This mismatch may lead to problems during the optimization process. Then, a natural question is: can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process ? In this paper, we provide our positive answer to this question. First, we conduct a formal study on the positive scaling operators which forms a transformation group, denoted as G. We show that the value of a path (i.e. the product of the weights along the path) in the neural network is invariant to positive scaling and prove that the value vector of all the paths is sufficient to represent the neural networks under mild conditions. Second, we show that one can identify some basis paths out of all the paths and prove that the linear span of their value vectors (denoted as G-space) is an invariant space with lower dimension under the positive scaling group. Finally, we design stochastic gradient descent algorithm in G-space (abbreviated as G-SGD) to optimize the value vector of the basis paths of neural networks with little extra cost by leveraging back-propagation. Our experiments show that G-SGD significantly outperforms the conventional SGD algorithm in optimizing ReLU networks on benchmark datasets.
Summary:
In prior deep learning papers it has been observed that ReLU networks are positively scale invariant due to the non-negative homogeneity property of the max(0, x) function. The present paper proposes to change the optimization procedure so that it is done in a space where this invariance is preserved (in contrast to the weight space, where it is not). To do so, the authors define the group G of positive scaling operators, and note that the "value of path" (product of weights along the path) is G-invariant and together with "activation status of paths" allows for the definition of an equivalence class. They then build a G-space which has fewer dimensions than the weight space, and proposed g-SGD to optimize the network in this space. In g-SGD gradients are computed normally, then projected to G-space via a sparse matrix in order to update the values of paths. The weights are then updated based on a "weight allocation method" that involves the inverse projection.
The authors conduct experiments on CIFAR-10 and -100 with a ResNet-34 and a similarly deep variant of VGG, showing significant benefits from G-SGD training in all cases. They also evaluate the performance of a simple MLP on Fashion-MNIST as a function of the invariant ratio H/m.
Comments:
The paper is organized well (with technical details of the proofs delegated to the appendix), and discusses the differences in comparison prior work. While evaluations on large-scale datasets would be helpful here, the present experiments suggest that optimization in G-space indeed consistently improves results, so the proposed method seems promising.
For completeness, it would be great to include Path-SGD results for the CIFAR experiments in Table 1, together with runtime information to highlight the benefits of the g-SGD algorithm and provide experimental proof that the computational overhead is indeed low.
If the authors are hoping for a wider adoption of the method, it would be helpful for the community to have the g-SGD code released within one of the standard deep learning frameworks.
Questions:
- Does it matter which weights are chosen as "free skeleton weights"? If these weights never get updated in the optimization procedure, could you please comment on the intuitive interpretation of the necessity of their presence?
- The text states that the computational overhead of the gradient of path norm is "very high". The Path-SGD paper proposes a method costing (B + 1)T, where the overhead an be small for large batches. It would be good to clarify this a bit in the present text.
- Is the advantage of g-SGD over SGD expected to be proportional to the invariant ratio for CNNs as well?