Learning Differentially Private Recurrent Language Models
We demonstrate that it is possible to train large recurrent language models with user-level differential privacy guarantees with only a negligible cost in predictive accuracy. Our work builds on recent advances in the training of deep networks on user-partitioned data and privacy accounting for stochastic gradient descent. In particular, we add user-level privacy protection to the federated averaging algorithm, which makes large step updates from user-level data. Our work demonstrates that given a dataset with a sufficiently large number of users (a requirement easily met by even small internet-scale datasets), achieving differential privacy comes at the cost of increased computation, rather than in decreased utility as in most prior work. We find that our private LSTM language models are quantitatively and qualitatively similar to un-noised models when trained on a large dataset.
 This paper extends the previous results on differentially private SGD to user-level differentially private recurrent language models. It experimentally shows that the proposed differentially private LSTM achieves comparable utility compared to the non-private model. The idea of training differentially private neural network is interesting and very important to the machine learning + differential privacy community. This work makes a pretty significant contribution to such topic. It adapts techniques from some previous work to address the difficulties in training language model and providing user-level privacy. The experiment shows good privacy and utility. The presentation of the paper can be improved a bit. For example, it might be better to have a preliminary section before Section2 introducing the original differentially private SGD algorithm with clipping, the original FedAvg and FedSGD, and moments accountant as well as privacy amplification; otherwise, it can be pretty difficult for readers who are not familiar with those concepts to fully understand the paper. Such introduction can also help readers understand the difficulty of adapting the original algorithms and appreciate the contributions of this work.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning Differentially Private Recurrent Language Models
We demonstrate that it is possible to train large recurrent language models with user-level differential privacy guarantees with only a negligible cost in predictive accuracy. Our work builds on recent advances in the training of deep networks on user-partitioned data and privacy accounting for stochastic gradient descent. In particular, we add user-level privacy protection to the federated averaging algorithm, which makes large step updates from user-level data. Our work demonstrates that given a dataset with a sufficiently large number of users (a requirement easily met by even small internet-scale datasets), achieving differential privacy comes at the cost of increased computation, rather than in decreased utility as in most prior work. We find that our private LSTM language models are quantitatively and qualitatively similar to un-noised models when trained on a large dataset.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper extends the previous results on differentially private SGD to user-level differentially private recurrent language models. It experimentally shows that the proposed differentially private LSTM achieves comparable utility compared to the non-private model. The idea of training differentially private neural network is interesting and very important to the machine learning + differential privacy community. This work makes a pretty significant contribution to such topic. It adapts techniques from some previous work to address the difficulties in training language model and providing user-level privacy. The experiment shows good privacy and utility. The presentation of the paper can be improved a bit. For example, it might be better to have a preliminary section before Section2 introducing the original differentially private SGD algorithm with clipping, the original FedAvg and FedSGD, and moments accountant as well as privacy amplification; otherwise, it can be pretty difficult for readers who are not familiar with those concepts to fully understand the paper. Such introduction can also help readers understand the difficulty of adapting the original algorithms and appreciate the contributions of this work.
assistant
 This paper extends the previous results on differentially private SGD to user-level differentially private recurrent language models. It experimentally shows that the proposed differentially private LSTM achieves comparable utility compared to the non-private model. The idea of training differentially private neural network is interesting and very important to the machine learning + differential privacy community. This work makes a pretty significant contribution to such topic. It adapts techniques from some previous work to address the difficulties in training language model and providing user-level privacy. The experiment shows good privacy and utility. The presentation of the paper can be improved a bit. For example, it might be better to have a preliminary section before Section2 introducing the original differentially private SGD algorithm with clipping, the original FedAvg and FedSGD, and moments accountant as well as privacy amplification; otherwise, it can be pretty difficult for readers who are not familiar with those concepts to fully understand the paper. Such introduction can also help readers understand the difficulty of adapting the original algorithms and appreciate the contributions of this work.