  Softmax loss is arguably one of the most popular losses to train CNN models
for image classification. However, recent works have exposed its limitation on
feature discriminability. This paper casts a new viewpoint on the weakness of
softmax loss. On the one hand, the CNN features learned using the softmax loss
are often inadequately discriminative. We hence introduce a soft-margin softmax
function to explicitly encourage the discrimination between different classes.
On the other hand, the learned classifier of softmax loss is weak. We propose
to assemble multiple these weak classifiers to a strong one, inspired by the
recognition that the diversity among weak classifiers is critical to a good
ensemble. To achieve the diversity, we adopt the Hilbert-Schmidt Independence
Criterion (HSIC). Considering these two aspects in one framework, we design a
novel loss, named as Ensemble soft-Margin Softmax (EM-Softmax). Extensive
experiments on benchmark datasets are conducted to show the superiority of our
design over the baseline softmax loss and several state-of-the-art
alternatives.

Computer Vision and Pattern Recognition