DropMax: Adaptive Variational Softmax
We propose DropMax, a stochastic version of softmax classifier which at each iteration drops non-target classes according to dropout probabilities adaptively decided for each instance. Specifically, we overlay binary masking variables over class output probabilities, which are input-adaptively learned via variational inference. This stochastic regularization has an effect of building an ensemble classifier out of exponentially many classifiers with different decision boundaries. Moreover, the learning of dropout rates for non-target classes on each instance allows the classifier to focus more on classification against the most confusing classes. We validate our model on multiple public datasets for classification, on which it obtains significantly improved accuracy over the regular softmax classifier and other baselines. Further analysis of the learned dropout probabilities shows that our model indeed selects confusing classes more often when it performs classification.
This paper proposes doing dropout in the output softmax layer during supervised training of neural net classifiers. The dropout probabilities are adapted per example. The probabilities are computed as a function of the penultimate layer of the classifier. So that layer is used to compute both : the logits, and the gating for those logits. This model combines ideas from adaptive dropout (Ba and Frey NIPS'13) and variational dropout (Kingma et al).  The key problem being solved is how to do inference to get the optimal dropout probabilities. There are several approximations made in the model in order to do this inference. If I understood correctly, the model essentially comes down to the following:  Two gating functions are being imposed to gate the logits. One gating function is computed by taking the penultimate layer and passing it through learned weights (W_theta). The other gating function is also computed by taking the penultimate layer and passing it through learned weights (W_phi), but the second gating also tries to take the output y into account by regressing to it using an auxiliary loss. Together the two gating functions produce a number in [0, 1] which is used as the retain probability for doing dropout in the output softmax.  Pros  - The proposed model leads to improvements in classification performance for   small datasets and tasks with fine categories.  Cons  - The model appears to be overly complicated. Is just one gating function that   includes the dependence on y (W_phi) really not sufficient ? It would be great to see empirical evidence that this complexity is useful.  - The deterministic attention baseline seems to be using gating derived from x   alone (without the second gating used in the proposed dropmax method which also regresses to y). It would be useful to have a clear comparison between dropmax and a deterministic version of dropmax.  - The second gating function is essentially doing the same task as   classification but with a unit-wise cross-entropy loss instead of a multiclass cross-entropy loss which the main classifier is using. The resulting model is a product of the two classifiers.  It is plausible that the reported gains are a result of this ensembling of 2 classifiers, and not the dropout regularization. A determinstic baseline (as suggested above) would help resolve this question.  Overall : The model is presented as a new variant of dropout derived using a somewhat complicated setup, when it can perhaps be more simply understood as ensembling a unit-wise logitic regression loss and the familiar softmax loss. It is not clear if the current presentation is the cleanest way to describe this model. However, the model is sound and leads to incremental improvements.