Recent years have witnessed two seemingly opposite developments of deep convolutional neural networks (CNNs). On the one hand, increasing the density of CNNs (e.g. by adding cross-layer connections) achieves higher accuracy. On the other hand, creating sparsity structures (e.g. through regularization and pruning methods) enjoys a more slim network structure. In this paper, we bridge these two by proposing a new network structure with locally dense yet externally sparse connections. This new structure uses dense modules as basic building blocks and then sparsely connects these modules via a novel algorithm during the training process. Experimental results demonstrate that the locally dense yet externally sparse structure could obtain competitive performance on benchmark tasks (CI-FAR10, CIFAR100,and ImageNet) while keeping the network structure slim.
Under the inspiration of bridging these two trends and search more efficient network structures, our paper explores methods which directly introduce sparsity into network structure thus avoid pruningafter-training strategy. In neural science, papers (e.g. (Betzel et al., 2017;Sztarker & Tomsic, 2011;Gazzaniga, 1988)) concentrating on the brain structure reveal that neuron connections in brain perform a locally dense but externally sparse property as paper (Betzel et al., 2017) shown, that the closer two regions are, the denser the connections between them will be. Visual cortex papers (Belliveau et al., 1991) show that while sensory information arrives at the cortex, it is fed up through hierarchy regions from primary area V1 up to higher areas such as V2, V4 and IT. Inside of each cor-tex layer, tightly packed pyramidal cells consist of basic locally dense structures in the brain. While our brain has been trained over time, internal densely connected modules will form a few long distance and cross-level connections to transfer information to higher hierarchy. Modular structures have shown vital importance in our brain behaviors such as specializing in information processing (Espinosa-Soto & Wagner, 2010), performing focal functions (Baldassano & Bassett, 2016), and supporting complex neural dynamics.
In this case, instead of creating local density by pruning redundancy on the trained model, we perform local density by prefixing untrained dense modules as tightly packed neuron cell in the human brain and let it evolving both the weights of itself and the sparse connection between them via training. Since DenseNet has reached theoretical densest connection status, we use a similar dense block structure with growth rate k, but only with very narrow channels in each block. The growth rate k (Huang et al. (2017)) is a hyper parameter in Densely Connected structures, which denotes growth rate of the input feature map scale when network goes deeper. Previous methods constructing neural modules with structural sparsity (e.g. (Xie et al., 2018;Szegedy et al., 2017)) are mostly empirically constructing the sparse connection between modules. To give more convincing guidance of forming sparse connections, we design a genetic training strategy to search an optimized connection matrix. This algorithm treats the connection matrix as the gene, and only reserves mutated individual with the best performance among others. Actually, this strategy consistently changes the input feature groups during training process, and by always counting new feature distribution in, this strategy could take similar effect as drop-out methods, thus make the model robust. Moreover, besides merely creating parallel connections between modules, our algorithm could create long-distance connections between input module and output module by a transit layer.
The experiment results demonstrate that evolving locally dense but externally sparse connections could maintain competitive performance on benchmark image datasets while using compared slim network structures. By comparison experiments, we reveal contribution proportion on the final performance of each specific connection, and by that give the principle of design sparse connections between dense modules. The main contribution of this paper is as follows:
• We enhance the hierarchical structure by utilizing the property of locally dense but externally sparse connections.
• Instead of empirically constructing module connections, we design an evolving training algorithm to search optimized connection matrix.
• We let each module choose output flow globally rather than simply creating parallel streams between modules so that the feature could flow to final layer through various depth.
• We give a detailed analysis of how different sparse connections and different module properties will contribute to the final performance. Moreover, We reveal contribution proportion on the final performance of each connection and each module by several contrast experiments, and by that give principle of design sparse connections between dense modules.
The authors present the interesting and important direction in searching better network architectures using the genetic algorithm. Performance on the benchmark datasets seems solid. Moreover, the learned insights described in Section 4.4 would be very helpful for many researchers.
However, the overall paper needs to be polished more. There are two many typos and errors that imply that the manuscript is not carefully polished. Explanations about some terms like growth rate, population, etc. are necessary for broader audience. 
More importantly, while some of step jumps in Figure 6~9 are suspicious, it turns out that all the step jumps happen at the same number of steps, which are identical to the change of learning rates described in Section 4.2. Thee clear explanation about that phenomena is required.
* Details
- Please represent the blocks (e.g. 1*1conv) better. Current representation is quite confusing to read. Maybe proper spacing and different style of fonts may help.
- In Page 5, "C_{m}ax" is a typo. It should be "C_{max}".
- Regarding the C_max, does sum(C_max) represent (D * W)^2 where D is the total depth and W is the total indicies in each layer? If so, specifying it will help. Otherwise, please explain its meaning clearly.
- In Figure 4(a), it would be better if we reuse M_{d,w} notation instead of Module {d_w}.
- Please briefly explain or provide references to the terms like "growth rate", "population", and "individuals". 
- Different mutations may favor different hyper-parameters. How the authors control the hyperparameters other than the number of epochs will be useful to know.
- Even though the sparse connection is enforced for some reasons, overfitting, variance, or any other benefits that slim structure can bring in has not been evaluated. They need to be presented to verify the hypothesis that the authors claim.