Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles
Many practical perception systems exist within larger processes that include interactions with users or additional components capable of evaluating the quality of predicted solutions. In these contexts, it is beneficial to provide these oracle mechanisms with multiple highly likely hypotheses rather than a single prediction. In this work, we pose the task of producing multiple outputs as a learning problem over an ensemble of deep networks -introducing a novel stochastic gradient descent based approach to minimize the loss with respect to an oracle. Our method is simple to implement, agnostic to both architecture and loss function, and parameter-free. Our approach achieves lower oracle error compared to existing methods on a wide range of tasks and deep architectures. We also show qualitatively that the diverse solutions produced often provide interpretable representations of task ambiguity.
The paper proposes a new method for training ensemble of Deep Neural Networks (DNN) (or any other gradient-based approach), by using an oracle (that knows with member of the ensemble is optimal for the current instance) over each data instances used for training, applying backprop only on the network with the output selected. The approach has the advantage of letting the networks to specialized themeselves, giving ensemble that are covering well all (or most) possible output of interest. Results are reported on a variety of domains, with overall great performances.
I think that's a pretty good paper. It presents well an interesting idea of training an ensemble of DNN with an oracle, which oracle is guiding backprop update to the net with the best output (according to the true label). It appears to enforce specialization quite well in a classification context. The overall presentation is really good and the experiments are done on several domains. Globally, a strong work that deserves to be published. I would have considered this paper as outstanding if the issue exploitation of ensemble of DNNs trained with the proposed approach would also been addressed. Currently, training is done with the oracle, but the use of the resulting ensemble is not developped further, outside saying it can provide a broad pool of possible results and that the approach is coherent with the top-k metric proposed for some problems (i.e., ImageNet). This is an important missing part for making the proposed approach fully usable in practice. Nevertheless, the contribution is there in the current stage, the missing part being relatively problem-specific. Fig. 2 presents pseudo-code that is in fact raster images embedded in the paper. Why not making the pseudo-code directly within the paper, or at least embed the pseudo-code figure as a vectorial image?