Deep Hyperspherical Learning
Convolution as inner product has been the founding basis of convolutional neural networks (CNNs) and the key to end-to-end visual representation learning. Benefiting from deeper architectures, recent CNNs have demonstrated increasingly strong representation abilities. Despite such improvement, the increased depth and larger parameter space have also led to challenges in properly training a network. In light of such challenges, we propose hyperspherical convolution (SphereConv), a novel learning framework that gives angular representations on hyperspheres. We introduce SphereNet, deep hyperspherical convolution networks that are distinct from conventional inner product based convolutional networks. In particular, SphereNet adopts SphereConv as its basic convolution operator and is supervised by generalized angular softmax loss -a natural loss formulation under SphereConv. We show that SphereNet can effectively encode discriminative representation and alleviate training difficulty, leading to easier optimization, faster convergence and comparable (even better) classification accuracy over convolutional counterparts. We also provide some theoretical insights for the advantages of learning on hyperspheres. In addition, we introduce the learnable SphereConv, i.e., a natural improvement over prefixed SphereConv, and SphereNorm, i.e., hyperspherical learning as a normalization method. Experiments have verified our conclusions.
This paper presents a novel architecture, SphereNet, which replaces the traditional dot product with geodesic distance as the convolution operators and fully-connected layers. SphereNet also regularizes the weights for softmax to be norm 1 for angular softmax. The results show that SphereNet can achieve superior performance in terms of accuracy and convergence rate as well as mitigating the vanishing/exploding gradients in deep networks.
Novelty: Replacing dot product similarity with angular similarity has widely existed in the deep learning literature. With that being said, most works focus on using angular similarity for Softmax or loss functions. This paper introduces spherical operation for convolution, which is novel in the literature as far as I know.
Significance: The spherical operations the paper introduces achieve faster convergence rate and better accuracy performance. It also mitigates the vanishing/exploding gradients brought by dot products, which is a long standing problem. It will be interesting to lots of people in ML community.
Improvement: I have several suggestions and questions as listed below:
- For angular Softmax, the bias is removed from the equation. However, the existence of bias is very important for calibrating the output, especially for imbalanced datasets. It would be great if the authors could add that back. Otherwise, it's better to try current model on an imbalanced dataset to measure the effect.
- In the first two subfigures of Figure 4, the baseline methods such as standard CNN actually converges faster at the very beginning. But there is a big accuracy drop in the middle. What is the root cause for that? The paper seems no giving any explanation.
- Though the experimental results show faster convergence rate in terms of accuracy vs iteration, it's unknown whether this observation still hold true for accuracy vs time. Traditional dot product operators leverage fast matrix multiplication libraries for speedup, while there is much less support for angular operators.
Minor on grammar:
- Line 216, "this problem will automatically solved" -> "be solved"
- Line 319, "The task can also viewed as a ..." -> "be viewed"