DISTRIBUTED FINE-TUNING OF LANGUAGE MODELS ON PRIVATE DATA
One of the big challenges in machine learning applications is that training data can be different from the real-world data faced by the algorithm. In language modeling, users' language (e.g. in private messaging) could change in a year and be completely different from what we observe in publicly available data. At the same time, public data can be used for obtaining general knowledge (i.e. general model of English). We study approaches to distributed fine-tuning of a general model on user private data with the additional requirements of maintaining the quality on the general data and minimization of communication costs. We propose a novel technique that significantly improves prediction quality on users' language compared to a general model and outperforms gradient compression methods in terms of communication efficiency. The proposed procedure is fast and leads to an almost 70% perplexity reduction and 8.7 percentage point improvement in keystroke saving rate on informal English texts. Finally, we propose an experimental framework for evaluating differential privacy of distributed training of language models and show that our approach has good privacy guarantees.
This paper deals with improving language models on mobile equipments
based on small portion of text that the user has ever input. For this
purpose, authors employed a linearly interpolated objectives between user
specific text and general English, and investigated which method (learning
without forgetting and random reheasal) and which interepolation works better.
Moreover, authors also look into privacy analysis to guarantee some level of
differential privacy is preserved.
Basically the motivation and method is good, the drawback of this paper is
its narrow scope and lack of necessary explanations. Reading the paper,
many questions arise in mind:
- The paper implicitly assumes that the statistics from all the users must
  be collected to improve "general English". Why is this necessary? Why not
  just using better enough basic English and the text of the target user?
- To achieve the goal above, huge data (not the "portion of the general English") should be communicated over the network. Is this really worth doing? If only
  "the portion of" general English must be communicated, why is it validated?
- For measuring performance, authors employ keystroke saving rate. For the
  purpose of mobile input, this is ok: but the use of language models will
  cover much different situation where keystrokes are not necessarily 
  available, such as speech recognition or machine translation. Since this 
  paper is concerned with a general methodology of language modeling, 
  perplexity improvement (or other criteria generally applicable) is also
  important.
- There are huge number of previous work on context dependent language models,
  let alone a mixture of general English and specific models. Are there any
  comparison with these previous efforts?
Finally, this research only relates to ICLR in that the language model employed
is LSTM: in other aspects, it easily and better fit to ordinary NLP conferences, such as EMNLP, NAACL or so. I would like to advise the authors to submit
this work to such conferences where it will be reviewed by more NLP experts.
Minor:
- t of $G_t$ in page 2 is not defined so far.
- What is "gr" in Section 2.2?