This paper presents a generalized approach to provide an individual optimal temperature trajectory to each class. Temperature scaling is an effective technique to control the smoothness of the distribution. Current implementations of temperature scaling assume either a fixed or manually-crafted dynamically changing schedule. This schedule is used to identify the shared optimal trajectory, which is applied to every class. However, our studies indicate that under the change of context, we can identify the corresponding individual optimal trajectory for each class. To this end, we propose contextual temperature, a mechanism that allows the temperature to be learned during training along with the remaining model parameters, over the context for each vocabulary. Experimental results confirm that contextual temperature significantly improves state-of-the-art language models. The proposed method achieves a perplexity of 55.31 and 62.89 on the test set of Penn Treebank and WikiText-2, respectively. Additionally, in-depth analyses indicate that (a) every vocabulary possesses its individual schedule of temperature, and (b) due to the increased context from every time step, the optimal trajectory drops in order to suppress the uncertainties.
This work proposes a learned and context dependent way to calculate the temperatures for the softmaxes. More specifically, a low-rank affine-transformation, taking the hidden state at the current step as input, is used to calculate scalar weighting for every token in the vocabulary. The method is very general, and can be used in combination with other techniques in tasks such as language modeling and text generation. Experiments on language modeling with Penn TreeBank and WikiText-2 show that the proposed method yields strong performance.
Overall I found the paper well-motivated and easy to follow. The empirical results are solid and strong. The analysis is also interesting. I vote for an acceptance, if the authors can polish the writing.
Details:
- Eq. 5. The temperature scalar for each token competes with each other, since they are calculated with a softmax (and then rescaled). Another way is to use, e.g., a sigmoid function. Can the authors explain the motivation behind the use of softmax?
- Another view of the proposed method is that it learns a context-dependent weighting of the tokens in the vocabulary, such that "important" tokens (those with smaller \tau) receive more gradient updates. Can the authors comment on this? Also, I don't see the thermodynamics connection and find calling the proposed method `temperature` a bit misleading. 
- Adding onto above. [1] discusses the low-rank bottleneck of using a single softmax. Since elementwise matrix product can blow up the rank, how do the authors think the proposed method can serve as a more efficient way to deal with the softmax bottleneck?
- Last but not least, the paper can be improved a lot if the authors can thoroughly polish the writing.
[1] Breaking the Softmax Bottleneck: A High-Rank RNN Language Model. https://arxiv.org/abs/1711.03953