Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing. Towards this end, we present a calibration-based approach to measure long-term discrepancies between a generative sequence model and the true distribution, and use these discrepancies to improve the model. Empirically, we show that state-of-the-art language models, including LSTMs and Transformers, are miscalibrated: the entropy rates of their generations drift dramatically upward over time. We then provide provable methods to mitigate this phenomenon. Furthermore, we show how this calibrationbased approach can also be used to measure the amount of memory that language models use for prediction.
This paper highlights and studies the problem of "entropy rate drift" for language models: the entropy rate of the language generated by a trained model is much higher than the entropy of ground truth sequences and this discrepancy worsen with the length of generation. The authors interestingly claim that the well-known lack of coherence in long-term model generations is due to this entropy rate drift. Valuably, the entropy rate drift is characterized mathematically. The authors propose a calibration method and prove that their method can interestingly reduce *both* the entropy rate drift and the perplexity of a miscalibrated model, *even though* they assume a rather simplistic model of miscalibration (one that leaks a small amount of mass to all the sequences of a given length). The author quantitatively show that their calibration method reduces the entropy rate drift and qualitatively show that their generations "make sense" in the long-term. As an auxiliary result, they show that a similar calibration method can be used to quantify the past information used by the model by upper-bounding the mutual information between the current prediction and the long-term past, given the short-term past, e.g. I(W_t | W<{t-\tau} | W_{t-\tau:t-1}).
I really enjoyed reading this paper and was really intrigued by the author's solution. However, in the current form, this paper is below the bar of acceptance due to some weaknesses: (i) rather strong assumption for the main derivation; (ii) lack of clarity and computational complexity of the proposed algorithms; (iii) weak experimental results and missing hint to current models / applications / concurring models. I would be more than happy to increase my score if the authors could kindly respond to the points below.
1) About the assumptions: in your theory, you show that the proposed solution is guaranteed to improve a particular constrained form of miscalibrated model P^\epsilon, in which a epsilon uniform distribution over all possible sentences of length K is added to the model distribution. This seems a rather strong assumption to me and bumps up to increasing the probability of each word by a small amount for each per-step conditional distribution estimated by the model.
1.1) Could you elaborate on why intuitively this assumption is something that happens in current models like GPT-2 ? Do you have a way of quantifying whether this assumption reasonably holds ?
2) About complexity:
2.1) What’s roughly the computational complexity of Algorithm 2 ?
2.2) Do you need to compute H(W_{t+1}|w_{<= t}) for all possible 1-step continuations w_t ? That seems quite expensive to me in the case of a large vocab. (e.g. sample a word, run forward one step and compute future entropy).
2.3) If this is correct, how to address this issue ? If I am missing something, it would be good to add to the paper some explicit considerations about the technical feasibility of the algorithm.
3) The amazing thing to me is that you reduce entropy-rate drift (although wrt a particular miscalibrated model, cf. 1) by re-minimizing cross-entropy using the ground-truth corpus. If you could give intuition for this , it would be great and make a much stronger paper !
3.1) Could you explain why intuitively this works ?
3.2) If \alpha is positive, Algorithm 2 is basically penalizing the model for producing words that lead to higher entropy in the future. Why this correlates to less cross-entropy with the data distribution ?
3.3) Why would calibrating a model that may not conform to the assumption 1) in general prefer to have an \alpha > 0 ?
4) About the experiments:
4.1) Where are the perplexities of the calibrated models ?
4.2) Are the perplexity improving as the theory would suggest ?
4.3) Is there any small , systematic human study that you could perform apart from just showing few examples of generations ?
4.4) Could you report some quantitative metrics of the generations from your models and the baselines, for example as computed in previous papers (https://arxiv.org/abs/1801.07736, https://arxiv.org/abs/1908.04319)
4.5) How do your generations change if you use beam search or arg-max, top-k sampling ? Does your method also help in preventing repetitions ?
Minor:
- I think you are missing a minus in Algorithm 2 inside the exponential.
====
Updated review:
I thank the authors for their answer. I think that overall this paper provides interesting insight on a hard problem.
I am raising my score and strongly hope that the authors add the considerations made in their response in the main paper: warning about complexity, optimal \alphas for each model and details.