We investigate in this paper the architecture of deep convolutional networks. Building on existing state of the art models, we propose a reconfiguration of the model parameters into several parallel branches at the global network level, with each branch being a standalone CNN. We show that this arrangement is an efficient way to significantly reduce the number of parameters while at the same time improving the performance. The use of branches brings an additional form of regularization. In addition to splitting the parameters into parallel branches, we propose a tighter coupling of these branches by averaging their log-probabilities. The tighter coupling favours the learning of better representations, even at the level of the individual branches, as compared to when each branch is trained independently. We refer to this branched architecture as "coupled ensembles". The approach is generic and can be applied to almost any neural network architecture. With coupled ensembles of DenseNet-BC and parameter budget of 25M, we obtain error rates of 2.92%, 15.68% and 1.50% on CIFAR-10, CIFAR-100 and SVHN respectively. For the same parameter budget, DenseNet-BC has an error rate of 3.46%, 17.18%, and 1.8% respectively. With ensembles of coupled ensembles, of DenseNet-BC networks, with 50M total parameters, we obtain error rates of 2.72%, 15.13% and 1.42% respectively on these tasks.
This paper presents a deep network architecture which processes data using multiple parallel branches and combines the posterior from these branches to compute the final scores; the network is trained in end-to-end, thus training the parallel branches jointly. Existing literature with branching architecture either employ a 2 stage training approach, training branches independently and then training the fusion network, or the branching is restricted to local regions (set of contiguous layers). In effect, this paper extends the existing literature suggesting end-to-end branching. While the technical novelty, as described in the paper, is relatively limited, the thorough experimentation together with detailed comparisons between intuitive ways to combine the output of the parallel branches is certainly valuable to the research community.
+ Paper is well written and easy to follow.
+ Proposed branching architecture clearly outperforms the baseline network (same number of parameters with a single branch) and thus offer yet another interesting choice while creating the network architecture for a problem
+ Detailed experiments to study and analyze the effect of various parameters including the number of branches as well as various architectures to combine the output of the parallel branches.
+ [Ease of implementation] Suggested architecture can be easily implemented using existing deep learning frameworks.
- Although joint end-to-end training of branches certainly brings value compared to independent training, but the increased resource requirements may limits the applicability to large benchmarks such as ImageNet. While authors suggests a way to circumvent such limitations by training branches on separate GPUs but this would still impose limits on the number of branches as well as its ease of implementation.
- Adding an overview figure of the architecture in the main paper (instead of supplementary) would be helpful.
- Branched architecture serve as a regularization by distributing the gradients across different branches; however this also suggests that early layers on the network across branches would be independent. It would helpful if authors would consider an alternate archiecture where early layers may be shared across branches, suggesting a delayed branching, with fusion at the final layer.
- One of the benefits of architectures such as DenseNet is their usefulness as a feature extractor (output of lower layers) which generalizes even to domain other that the dataset; the branched architecture could potentially diminish this benefit.
Minor edits: Page 1. 'significantly match and improve' => 'either match or improve'
Additional notes:
- It would interesting to compare this approach with a conditional training pipeline that sequentially adds branches, keeping the previous branches fixed. This may offer as a trade-off between benefits of joint training of branches vs being able to train deep models with several branches.