Ensemble learning is a method of combining multiple trained models to improve model accuracy. We propose the usage of such methods, specifically ensemble average, inside Convolutional Neural Network (CNN) architectures by replacing the single convolutional layers with Inner Average Ensembles (IEA) of multiple convolutional layers. Empirical results on different benchmarking datasets show that CNN models using IEA outperform those with regular convolutional layers and advances the state of art. A visual and a similarity score analysis of the features generated from IEA explains why it boosts the model performance.
IEA proposes to use multiple "parallel" convolution groups, which are then averaged to improve performance.
This fundamental idea of ensembles combined with simple functions has been explored in detail in Maxout (Goodfellow et.  al., https://arxiv.org/abs/1302.4389) in the context of learning activation functions, and greater integration with dropout regularization.
Under the lens of comparison to Maxout (which should be cited, and is a key comparison point for this work), a number of questions emerge. Does IEA also work for feedforward layers? Does IEA give any performance improvement or have some fundamental synergy with the regularizers used here? Is the performance boost greater than simply using an ensemble of m networks directly (resulting in the equivalent number of parameters overall)? The choice of the mean here seems insufficient for creating the types of complexity in activation which are normally desirable for neural networks, so some description of why a simple mean is a good choice would be beneficial since many, many other functions are possible.
Crucially Maxout seems much too close to this work, and I would like to see an indepth comparison (since it appears to be use of mean() instead of max() is the primary difference). I would also significantly reduce the claims of novelty, such as "We introduce the usage of such methods, specifically ensemble average inside Convolutional Neural Networks (CNNs) architectures." in the abstract, given that this is the exact idea explored in other work including followups to Maxout.
For example, MNIST performance here matches Maxout (.45% for both, but Maxout uses techniques known in 2013). CIFAR-10 results are better, but again Maxout first appeared 5 years ago. There are more recent followups that continued on the line of work first shown in Maxout, and there should be some greater comparison and literature review on these papers. The CIFAR-10 baseline numbers are not ideal, and since IEA is basically "plug and play" in existing architectures, starting from one of these settings instead (such as Wide ResNet https://arxiv.org/abs/1605.07146) and showing a boost would be a stronger indication that this method actually improves results. In addition, there are a number of non-image settings where CNNs are used (text or audio), and showing this idea works on multiple domains would also be good.
There seems to be a similarity between ResNet and this method - specifically assuming the residual pathway is convolution with an identity activation, the summation that combines the two pathways bears a similarity to IEA. With multiple combined paths (as in Dense ResNet) this equivalence seems stronger still. A discussion of this comparison in greater detail, or even derivation of IEA as a special setting or extension of ResNet (coupled with stronger performance on the datasets) would help ground the work in prior publication.
The section on visualization and inspection of IEA features seems interesting, but too brief. A greater exploration of this, and possible reduction or removal of the ensemble selection section (which didn't have a clear contribution to the message of the paper, in my opinion) would strengthen the work - and again, comparisons to activations learned by Maxout and followups would make this inspection much stronger.
My key concerns here are on relation to past work, greater comparison to closely related methods, and improvement of baselines results. Given the close similarity of this work to Maxout and others, a much stronger indication of the benefits and improvements of IEA seems necessary to prove out the concepts here.