We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections. Each Crescendo block contains independent convolution paths with increased depths. The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks. In experiments, CrescendoNet with only 15 layers outperforms almost all networks without residual connections on benchmark datasets, CIFAR10, CIFAR100, and SVHN. Given sufficient amount of data as in SVHN dataset, CrescendoNet with 15 layers and 4.1M parameters can match the performance of DenseNet-BC with 250 layers and 15.3M parameters. CrescendoNet provides a new way to construct high performance deep convolutional neural networks with simple network architecture. Moreover, through investigating the behavior and performance of subnetworks in CrescendoNet, we note that the high performance of CrescendoNet may come from its implicit ensemble behavior. Furthermore, the independence between paths in CrescendoNet allows us to introduce a new path-wise training procedure, which can reduce the memory needed for training.
The paper presents a new CNN architecture: CrescendoNet. It does not have skip connections yet performs quite well.
Overall, I think the contributions of this paper are too marginal for acceptance in a top tier conference.
The architecture is competitive on SVHN and CIFAR 10 but not on CIFAR 100. The performance is not strong enough to warrant acceptance by itself.
FractalNets amd DiracNets (https://arxiv.org/pdf/1706.00388.pdf) have demonstrated that it is possible to train deep networks without skip connections and achieve high performance. While CrescendoNet seems to slightly outperform FractalNet in the experiments conducted, it is itself outperformed by DiracNet. Hence, CrescendoNet does not have the best performance among skip connection free networks.
You claim that FractalNet shows no ensemble behavior. This is clearly not true because FractalNet has ensembling directly built in, i.e. different paths in the network are explicitly averaged. If averaging paths leads to ensembling in CrescendoNet, it leads to ensembling in FractalNet. While the longest path in FractalNet is stronger than the other members of the ensemble, it is nevertheless an ensemble. Besides, as Veit showed, ResNet also shows ensemble behavior. Hence, using ensembling in deep networks is not a significant contribution.
The authors claim that "Through our analysis and experiments, we note that the implicit ensemble behavior of CrescendoNet leads to high performance". I don't think the experiments show that ensemble behavior leads to high performance. Just because a network performs averaging of different paths and individual paths perform worse than sets of paths doesn't imply that ensembling as a mechanism is in fact the cause of the performance of the entire architecture. Similary, you say "On the other hand, the ensemble model can explain the performance improvement easily." Veit et al only claimed that ensembling is a feature of ResNet, but they did not claim that this was the cause of the performance of ResNet.
Path-wise training is not original enough or indeed different enough from drop-path to count as a major contribution.
You claim that the number of layers "increase exponentially" in FractalNet. This is misleading. The number of layers increases exponentially in the number of paths, but not in the depth of the network. In fact, the number of layers is linear in the depth of the network. Since depth is the meaningful quantity here, CrescendoNet does not have an advantage over FractalNet in terms of layer number. Also, it is always possible to simply add more paths to FractalNet if desired without increasing depth. Instead of using 1 long paths, one can simply use 2, 3, 4 etc. While this is not explicitly mentioned in the FractalNet paper, it clearly would not break the design principle of FractalNet which is to train a path of multiple layers by ensembling it with a path of fewer layers. CrescendoNets do not extend beyond this design principle.
You say that "First, path-wise training procedure significantly reduces the memory requirements for convolutional layers, which constitutes the major memory cost for training CNNs. For example, the higher bound of the memory required can be reduced to about 40% for a Crescendo block with 4 paths where interval = 1." This is misleading, as you need to store the weights of all convolutional layers to compute the forward pass and the majority of the weights of all convolutional layers to compute the backward pass, no matter how many weights you intend to update. In a response to a question I posed, you mentioned that we you meant was "we use about 40% memory for the gradient computation and storage". Fair enough, but "gradient computation and storage" is not mentioned in the paper. Also, the reduction to 40% does not apply e.g. to vanilla SGD because the computed gradient can be immediately added to the weights and does not need to be stored or combined with e.g. a stored momentum term.
Finally, nowhere in the paper do you mention which nonlinearities you used or if you used any at all. In future revisions, this should be rectified.
While I can definitely imagine that your network architecture is well-designed and a good choice for image classification tasks, there is a very saturated market of papers proposing various architectures for CIFAR-10 and related datasets. To be accepted to ICLR, either outstanding performance or truly novel design principles are required.