We propose DropMax, a stochastic version of softmax classifier which at each iteration drops non-target classes with some probability, for each instance. Specifically, we overlay binary masking variables over class output probabilities, which are learned based on the input via regularized variational inference. This stochastic regularization has an effect of building an ensemble classifier out of exponential number of classifiers with different decision boundaries. Moreover, the learning of dropout probabilities for non-target classes on each instance allows the classifier to focus more on classification against the most confusing classes. We validate our model on multiple public datasets for classification, on which it obtains improved accuracy over regular softmax classifier and other baselines. Further analysis of the learned dropout masks shows that our model indeed selects confusing classes more often when it performs classification.
Pros
- The proposed model is a nice way of multiplicatively combining two features :
  one which determines which classes to pay attention to, and other that
provides useful features for discrimination.
- The adaptive component seems to provide improvements for small dataset sizes
  and large number of classes.
Cons
- "One can easily see that if o_t(x; w) = 0, then class t becomes neutral in the
  classification and the gradients are not back-propagated from it." : This does
not seem to be true. Even if the logits are zero, the class would have a
non-zero probability and would receive gradients. Do the authors mean
exp(o_t(x;w)) = 0 ?
- Related to the above, it should be clarified what is meant by dropping a
  class. Is its logit set to zero or -\infty ? Excluding a class from the
softmax is equivalent to having a logit of -\infty, not zero. However, from the
equations in the paper it seems that the logit is set to zero. This would not
result in excluding the unit. The overall effect would just be to raise the
magnitude of logits across the entire softmax.
- It seems that the model benefits from at least two separate effects - one is
  the attention mechanism provided by the sigmoids, and the other is the
stochasticity during training. Presently, it is not clear if only one of the
components is providing most of the benefits, or if both things are useful. It
would be great to compare this model to a non-stochastic one which just has the
multiplicative effects applied in a deterministic way (during both training and
testing).
- The objective of the attention mechanism that sets the dropout mask seems to
  be the same as the primary objective of classifying the input, and the
attention mechanism is prevented from solving the task by adding an extra
entropy regularization. It would be useful to explain more why this is needed.
Would it not be fine if the attention mechanism did a perfect job of selecting
the class ?
Quality
The paper makes relevant comparisons and is overall well-motivated. However,
some aspects of the paper can be improved by adding more explanations.
Clarity
Some crucial aspects of the paper are unclear as mentioned above.
Originality
The main contribution of the paper is similar to multiplicative gating. The
added stochasticity and the model ensembling interpretation is probably novel.
However, experiments are insufficient to determine whether it is this novelty
that contributes to improved performance or just the gating.
Significance
This paper makes incremental improvements and would be of moderate interest to
the machine learning community.
Typos :
- In Eq 3, the numerator has z_t. Should that be z_y ?
- In Eq 5, the denominator has z_y. Should that be z_t ?