Collaborative personalization, such as through learned user representations (embeddings), can improve the prediction accuracy of neural-network-based models significantly. We propose Federated User Representation Learning (FURL), a simple, scalable, privacy-preserving and resource-efficient way to utilize existing neural personalization techniques in the Federated Learning (FL) setting. FURL divides model parameters into federated and private parameters. Private parameters, such as private user embeddings, are trained locally, but unlike federated parameters, they are not transferred to or averaged on the server. We show theoretically that this parameter split does not affect training for most model personalization approaches. Storing user embeddings locally not only preserves user privacy, but also improves memory locality of personalization compared to onserver training. We evaluate FURL on two datasets, demonstrating a significant improvement in model quality with 8% and 51% performance increases, and approximately the same level of performance as centralized training with only 0% and 4% reductions. Furthermore, we show that user embeddings learned in FL and the centralized setting have a very similar structure, indicating that FURL can learn collaboratively through the shared parameters while preserving user privacy.
In this paper, the authors propose using federated learning (FL) to train personalized models, which improves the scalability and privacy preservation of the existing personalization techniques. The empirical results show good performance.
However, in general, I think the contribution is limited. The reasons are as follows:
1. The proposed algorithm, FURL, is a direct and simple combination of personalized model and FL. Although the authors claim that there is significant improvement in the performance, such improvement comes from the personalization. And, the personalization itself is not a novel thing (I think the personalized model used in this paper is similar to [1] or some other references. Please correct me if the personalized model used in this paper is new, since I'm not an expert in personalization.) Thus, in general, this paper simply use FL to replace fully synchronous SGD in the training of the personalized models. All the benefits claimed in the introduction, including scalability, privacy preservation, and improvement of performance, come from either vanilla personalization or vanilla FL. I fail to find any new contribution in this combination.
2. The authors emphasize a lot on the "independent aggregation constraint". Although it sounds like such constraint is designed especially for FL + personalization, it is actually a feature only for personalization, which has nothing to do with FL. Note that when doing inference/prediction, each user uses his/her own private part of the model. Different users' private part of models will never affect each other. It is equivalent to training a global model, which concatenates the private parts of models into a big model, and each user update the global model in a sparse manner. Thus, we can also train such personalized model with fully synchronous SGD with sparse gradients, which also does not synchronize the private parts. The private part is never shared by different users, no matter trained by fully synchronous SGD or FL.
------------
References
[1] Jaech, Aaron, and Mari Ostendorf. "Personalized language model for query auto-completion." arXiv preprint arXiv:1804.09661 (2018).