Sigsoftmax: Reanalysis of the Softmax Bottleneck
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an output activation function for breaking the softmax bottleneck without additional parameters. We re-analyze the softmax bottleneck from the perspective of the output set of log-softmax and identify the cause of the softmax bottleneck. On the basis of this analysis, we propose sigsoftmax, which is composed of a multiplication of an exponential function and sigmoid function. Sigsoftmax can break the softmax bottleneck. The experiments on language modeling demonstrate that sigsoftmax and mixture of sigsoftmax outperform softmax and mixture of softmax, respectively.
The paper analyzes ability of the soft-max, if used as the output activation function in NN, to approximate posterior distribution. The problem is translated to the study of the rank of the matrices contating the log-probabilities computed by the analyzed activation layer. It is shown that the soft-max does not increases the rank of the input response matrix (i.e. output of the penultimate layer) by more than 1. The authors propose to replace soft-max by the so called sigsoftmax (i.e. product of sigmoid and soft-max functions). It is shown that the rank of sigsoftmax matrix is not less the rank of soft-max. The other outcome of the paper is an empirical evaluation of several activation functions on two language modeling tasks.  The provided theorems studying rank of response matrices obtained by various activation functions have their values per se. However, a practical impact of the results and their relevance to the studied language modeling problem are not fully clear from the paper.  In particular, the presented results characterize the rank of the matrix containing the output responses. The rank itself is clearly a proxy for true objective which is not defined in the paper but according to the experiments it seems to be the perplexity of the learned model. The question is how is the rank (i.e. quantity analyzed in the paper) related to the true objective (i.e. the perplexity). E.g. if some activation function produces response matrices with slightly higher ranks (compared to other activation function) does it imply that the corresponding NN model will have lower perplexity ? This question is not discussed in the paper.  The provided thorems suggest that the proposed sigsoftmax is only slightly better than the softmax. In particular, according Theorem 4 the sigsoftmax is proved not to be worse, in terms of the rank, than the softmax.  The empirical study carried on two datasets shows improvements of the sigsoftmax function over softmax by approximately 1 unit. It is not clear if this improvement is statistically relevant because: i) the number of datasets is limited and ii) the confidence of the estimate is not given (the reported STD is not a good measure of the confidence because it measures variation of the results w.r.t. network initialization according to line 268-269).   A pragmatic way to increase expressiveness of NN with the soft-max (and other functions) would to simply increase dimension $d$ of the hidden (penultimate) layer. It would increase the number of parameters, however, the same happens if one uses the mixture of soft-max (MOS). It is not clear why such obvious baseline was not evaluated in the paper.