SplineNets: Continuous Neural Decision Graphs
We present SplineNets, a practical and novel approach for using conditioning in convolutional neural networks (CNNs). SplineNets are continuous generalizations of neural decision graphs, and they can dramatically reduce runtime complexity and computation costs of CNNs, while maintaining or even increasing accuracy. Functions of SplineNets are both dynamic (i.e., conditioned on the input) and hierarchical (i.e., conditioned on the computational path). SplineNets employ a unified loss function with a desired level of smoothness over both the network and decision parameters, while allowing for sparse activation of a subset of nodes for individual samples. In particular, we embed infinitely many function weights (e.g. filters) on smooth, low dimensional manifolds parameterized by compact B-splines, which are indexed by a position parameter. Instead of sampling from a categorical distribution to pick a branch, samples choose a continuous position to pick a function weight. We further show that by maximizing the mutual information between spline positions and class labels, the network can be optimally utilized and specialized for classification tasks. Experiments show that our approach can significantly increase the accuracy of ResNets with negligible cost in speed, matching the precision of a 110 level ResNet with a 32 level SplineNet.
The paper presents SplineNets that reformulate CNNs as neural decision graph using B-splines. It's comprised of four technical contributions, i.e., embedding trick, general formulation for a neural decision graph, a loss function of utilizing and specializing splines and a differentiable quantization method.  The idea is novel and sensible. It integrates the classic splines into convolutional neural networks which might be valuable for both of theoretical and practical aspects. However, the paper is difficult to read and follow. It is unclear how they implement in practice. Some equations and symbols need to be clarified, e.g. the symbols in equation 7-10, and d indicates degree of the polynomials in line 115 but hyperparameter delta in line 235.  I'm confused with several points.  a) what's the role of theta? Is the phi (i.e. D(x; theta) in line 137) analogous to attention (or gating) mechanism?  b)  Does the k/K in line 233 denote the knot number? Does the degree of the polynomial default to be k-1? Or is a fixed degree used? If larger K indicates higher-order smoothness in F-SN, does it mean linear spline works best as J-SN always works better than F-SN. b) how to demonstrate the dynamic (conditioned on the input) introduced by the method?  d) More experimental details need to be provided and clarified, e.g. optimization method, important hyperparameters for training, filter sizes.  I like the idea, but the current draft need to be improved.  After reading the comments of other reviewers and rebuttal, I think it's a good work. I hope the final manuscript could include the updates present in the rebuttal.