title: Contextual Temperature for Language Modeling
abstract: Temperature scaling has been widely used to improve performance for NLP tasks that utilize Softmax decision layer. Current practices in using temperature either assume a ﬁxed value or a dynamically changing temperature but with a ﬁxed schedule. Little has been known on an optimal trajectory of temperature that can change with the context. In this paper, we propose contextual temperature, a mechanism that allows temperatures to change over the context for each vocabulary, and to co-adopt with model parameters during training. Experimental results illustrated that contextual temperature improves over state-of-the-art language models signiﬁcantly. Our model CT-MoS achieved a perplexity of 55.31 in the test set of Penn Treebank and a perplexity of 62.89 in the test set of WikiText-2. The in-depth analysis showed that the behavior of temperature schedule varies dramatically by vocabulary. The optimal temperature trajectory drops as the context becomes longer to suppress uncertainties in language modeling. These evidence further justiﬁed the need for contextual temperature and explained its performance advantage over ﬁxed temperature or scheduling.
authors: Pei-Hsin Wang Sheng-Iou Hsieh Shieh-Chieh Chang Jia-Yu Pan Yu-Ting Chen Wei Wei Da-Cheng Juan
hIndex: Jia-Yu Pan 3