Unbounded cache model for online language modeling with open vocabulary
Recently, continuous cache models were proposed as extensions to recurrent neural network language models, to adapt their predictions to local changes in the data distribution. These models only capture the local context, of up to a few thousands tokens. In this paper, we propose an extension of continuous cache models, which can scale to larger contexts. In particular, we use a large scale non-parametric memory component that stores all the hidden activations seen in the past. We leverage recent advances in approximate nearest neighbor search and quantization algorithms to store millions of representations while searching them efficiently. We conduct extensive experiments showing that our approach significantly improves the perplexity of pre-trained language models on new distributions, and can scale efficiently to much larger contexts than previously proposed local cache models.
This paper discusses an extensions to the recently proposed continuous cache models by Grave et al. The authors propose a continuous cache model that is unbounded, hence can take into account events that happened an indefinitely long time ago. While interesting, the paper fails to provide good experimental evidence of its merits. Its main statement is that this model is better than Grave et al., but then does not compare with it. It only seems to compare with cache models from the nineties (Kuhn et al.), although that is not even clear as they spend only one line (line 206) discussing the models they compare with. "the static model interpolated with the unigram probability distribution observed up to time t" does sound like Kuhn et al. and is definitely not Grave et al.
The authors also mention the importance of large vocabularies, yet fail to specify the vocabulary size for any of their experiments. I also don't understand why all the datasets were lowercased if large vocabularies are the target? This (and the above) is a real pity, because sections 1-3 looked very promising. We advise the authors to spend some more care on the experiments section to make this paper publishable.
Minor comments:
* line 5: stores -> store
* line 13: twice "be"
* line 30: speach -> speech
* line 31: "THE model"
* line 59: "assumptions ABOUT"
* line 72: no comma after "and"
* line 79: algorithm -> algorithmS
* line 79: approach -> approachES
* line 97: non-parameteric -> non-parametric
* line 99: "THE nineties"
* line 110: "aN update rule"
* line 127: Khun -> Kuhn
* line 197: motivationS
* line 197: "adapt to A changing distribution"
* line 211: "time steps"
* line 211: adaptative -> adaptive
* table 2: the caption mentions that the model is trained on news 2007, but in fact this varies throughout the table?
* line 216: interest -> interested
* line 235: millions -> million
* line 267: experimentS
* line 269: where do these percentages come from? they seem wrong...
* line 281: "THE static model"
* line 283: Set