Deep Neural Nets with Interpolating Function as Output Activation
We replace the output layer of deep neural nets, typically the softmax function, by a novel interpolating function. And we propose end-to-end training and testing algorithms for this new architecture. Compared to classical neural nets with softmax function as output activation, the surrogate with interpolating function as output activation combines advantages of both deep and manifold learning. The new framework demonstrates the following major advantages: First, it is better applicable to the case with insufficient training data. Second, it significantly improves the generalization accuracy on a wide variety of networks. The algorithm is implemented in PyTorch, and the code is available at https://github.com/ BaoWangMath/DNN-DataDependentActivation.
This paper develops a new data-dependent output activation function base on interpolation function. It is a nonparametric model based on a subset of training data. The activation function  is defined in an implicit manner by solving a set of linear equations. Therefore, it cannot be solved directly by backpropagation. Instead it proposes an auxiliary network with linear output to approximate the gradient. Experiment results show that it can improve the generalization ability on a variety of networks. And it also improves the performance and robustness when the data size is small. The strength of the paper is that the proposed algorithm consistently improves performance by combining nonparametric method with DNN. More importantly, it improves the robustness of DNN for small data case. On the other hand, the weakness of the paper is that the complexity of using nonparametric approach is usually high and it requires storage of the template samples. It seems that the algorithm need to store a lot of template samples in experiments (half of the training set)? It is not clear how the number of required template samples scale with problem size. Furthermore, the argument in lines 96-102 for developing the approximate approach for training the model is a bit heuristic.   Comments: *If we are using so many template data samples, why can’t we learn a parametric interpolation function by optimizing the objective function (1)?  *Figure 1(b) is not correct. The groundtruth label Y should not be used as input to the model during testing time. *In line 75, “is need to be” should be “need to be”. *In the equation above line 93, it is not explained how \hat{X}^{te} is obtained. From Algorithm 1, it can be conjectured that it is obtained by passing X^{te} through DNN and buffer layer. It should be clarified after this equation in the main text. *The discussion in Section 3 is not clear. And the discussion about (5) is unclear and cannot be understood. *In the caption of Figure 3, “plot the generation” should be “plot the generalization”. *In line 179, “(1-400 and 406-805 epochs corresponds to linear activation)”. I think 406-805 should be the part after switching to WNLL? *In lin 199, why mean is used? Why not use mean and standard deviation? *In line 232, “tTis” should be “This”.