Learning the Number of Neurons in Deep Networks
Nowadays, the number of layers and of neurons in each layer of a deep network are typically set manually. While very deep and wide networks have proven effective in general, they come at a high memory and computation cost, thus making them impractical for constrained platforms. These networks, however, are known to have many redundant parameters, and could thus, in principle, be replaced by more compact architectures. In this paper, we introduce an approach to automatically determining the number of neurons in each layer of a deep network during learning. To this end, we propose to make use of a group sparsity regularizer on the parameters of the network, where each group is defined to act on a single neuron. Starting from an overcomplete network, we show that our approach can reduce the number of parameters by up to 80% while retaining or even improving the network accuracy.
Group lasso regularization was added to remove neurons for deep model selection. Reported experimental results on a couple of deep models showed the expected model reduction with comparable prediction accuracy on several benchmark datasets.  
Stochastic proximal gradient descent was presented to solve the deep model training with the proposed model selection based on group lasso regularization. However, the implementation details were not presented, especially regarding the iterative updates for the "loss-based gradient  step". From the context, it seems that the iterative updates in the original deep model training procedure were simply modified based on Eq. 5. If that is the case, the claim of the generality of the method for "general deep networks" as it will depend on the optimization procedure of the deep model training.   If the motivation of the presented work is to remove redundant neurons. it appears natural to compare the proposed method with the dropout method to see the performance. Without that, it is difficult to evaluate the significance and advantages of the method.   The presented performance evaluation is problematic. It is somewhat strange that both the performance evaluation results were done based on some modifications of the original deep models, either removing the first two layers (BnetC) or changing maxout to maxpooling (Dec3). Why was the comparison not done to the original model without changes, especially considering the reported decrease of top-1 accuracy in Table 1? It is not described how "accuracy gap" was computed. If it is simply the top-1 accuracy difference, it does not seem to be consistent based on the tables in Figs 3 and 4.  More critically, as typically done in sparse models, the sensitivity of the method with respect to penalty coefficients should be analyzed, especially with the complicated loss function in deep models. There are also many confusing places in experiments. For example, in Table 1, what is the difference between BnetC and "Ours-BnetC" and for other similar pairs? What does "S-GS" stand for (SGL?) in the table of Fig. 4?   There are also typos throughout the paper. For example, in line 275: "zeroed-our" should be "zeroed-out"; in line 293, "were" should be "where"?