Skip connections are increasingly utilized by deep neural networks to improve accuracy and cost-efficiency. In particular, the recent DenseNet is efficient in computation and parameters, and achieves state-of-the-art predictions by directly connecting each feature layer to all previous ones. However, DenseNet's extreme connectivity pattern may hinder its scalability to high depths, and in applications like fully convolutional networks, full DenseNet connections are prohibitively expensive. This work first experimentally shows that one key advantage of skip connections is to have short distances among feature layers during backpropagation. Specifically, using a fixed number of skip connections, the connection patterns with shorter backpropagation distance among layers have more accurate predictions. Following this insight, we propose a connection template, Log-DenseNet, which, in comparison to DenseNet, only slightly increases the backpropagation distances among layers from 1 to (1+log 2 L), but uses only L log 2 L total connections instead of O(L 2 ). Hence, Log-DenseNets are easier to scale than DenseNets, and no longer require careful GPU memory management. We demonstrate the effectiveness of our design principle through ablation studies and by showing better performance than DenseNets on tabula rasa semantic segmentation, and competitive results on visual recognition.
This paper investigates how to impose layer-wise connections in DenseNets most efficiently. The authors propose a connection-pattern, which connects layer i to layer i-2^k, k=0,1,2... The authors also propose maximum backpropgation distance (MBD) for measuring the fluency of gradient flow in the network, and justify the Log-DenseNet's advantage in this framework. Empirically, the author demonstrates the effectiveness of Log-DenseNet by comparing it with two other intuitive connection patterns on CIFAR datasets. Log-DenseNet also improves on FC-DenseNet, where the connection budget is the bottleneck because the feature maps are of high resolutions.
Strengths:
1. Generally, DenseNet is memory-hungry if the connection is dense, and it is worth studying how to sparsify a DenseNet. By showing the improvements on FC-DenseNet, Log-DenseNet demonstrates good potential on tasks which require upsampling of feature maps. 
2. The ablation experiments are well-designed and the visualizations of connectivity pattern are clear.
Weakness:
1. Adding a comparison with Log-DenseNet and vanilla DenseNet in the Table 2 experiment would make the paper stronger. Also, the NearestHalfAndLog pattern is not used in any latter visual recognition experiments, so I think it's better to just compare LogDenseNet with the two baselines instead. Despite there are CIFAR experiments on Log-DenseNet in latter sections, including results here would be easier to follow.
2. I would like to see the a comparison with the DenseNet-BC in the segmentation and CIFAR classification tasks, which uses 1x1 conv layers to reduce the number of channels. It should be interesting to study whether it is possible to further sparsify DenseNet-BC, as it has much higher efficiency.
3. The improvement of efficiency on classifications task is not that significant.