To continuously improve quality and reflect changes in data, machine learningbased services have to regularly re-train and update their core models. In the setting of language models, we show that a comparative analysis of model snapshots before and after an update can reveal a surprising amount of detailed information about the changes in the data used for training before and after the update. We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.
This paper provides an empirical evaluation of the privacy implications of releasing updated versions of language models. The authors show how access to two sequential snapshots of a trained language model can reveal highly specific information about the content of the data used to update the model, even when that data is in-distribution.
The paper contains easy to understand, concrete experiments and results, but seems altogether a little underdeveloped. The methodology is sound, but the synthetic experiments around which much of the paper is based may not be sufficiently novel and give little indication of broader implications. It would have been more convincing if these results replicated with two splits of the same dataset, rather than identical datasets with one augmented by canary tokens. 
 
The qualitative evaluation of subject-specific updates is also not sufficiently informative. It would have been useful to define a specific attack and see under what circumstances such an attack would succeed. In the current results, I am not convinced that any of the phrases in Table 3 represent a privacy violation. 
The differential privacy experiment seems to be missing many details: what dataset was this trained on? Are the accuracy values for the training set or a separate testing set? Other works have shown that it is possible to train a differentially private language model without large sacrifices in accuracy, so it would be helpful to know what differentiates this experiment. 
I would also note that the motivation, a predictive keyboard, is not a situation in which maximizing accuracy is generally desirable: users tend to find this creepy rather than helpful.
This is a nice idea but would benefit from some more polishing and more extensive testing.