In this research, we present a novel learning scheme called network iterative learning for deep neural networks. Different from traditional optimization algorithms that usually optimize directly on a static objective function, we propose in this work to optimize a dynamic objective function in an iterative fashion capable of adapting its function form when being optimized. The optimization is implemented as a series of intermediate neural net functions that is able to dynamically grow into the targeted neural net objective function. This is done via network morphism so that the network knowledge is fully preserved with each network growth. Experimental results demonstrate that the proposed network iterative learning scheme is able to significantly alleviate the degradation problem. Its effectiveness is verified on diverse benchmark datasets.
This submission develops a learning scheme for training deep neural networks with adoption of network morphism (Wei et al., 2016), which optimizes a dynamic objective function in an iterative fashion capable of adapting its function form when being optimized, instead of directly optimizing a static objective function. Overall, the idea looks interesting and the manuscript is well-written. The shown experimental results should be able to validate the effectiveness of the learning scheme to some extent.
It would be more convincing to include the performance evaluation of the learning scheme in some representative applications, since the optimality of the training objective function is not necessarily the same as that of the trained network in the application of interest.
Below are two minor issues:
- In page 2, it is stated that Fig. 2(e) illustrates the idea of the proposed network iterative learning scheme for deep neural networks based on network morphism. However, the idea seems not clear from Fig. 2(e).
- In page 4, “such network iterative learning process” should be “such a network iterative learning process”.