Communication Efficient Distributed Machine Learning with the Parameter Server
This paper describes a third-generation parameter server framework for distributed machine learning. This framework offers two relaxations to balance system performance and algorithm efficiency. We propose a new algorithm that takes advantage of this framework to solve non-convex non-smooth problems with convergence guarantees. We present an in-depth analysis of two large scale machine learning problems ranging from 1 -regularized logistic regression on CPUs to reconstruction ICA on GPUs, using 636TB of real data with hundreds of billions of samples and dimensions. We demonstrate using these examples that the parameter server framework is an effective and straightforward way to scale machine learning to larger problems and systems than have been previously achieved.
This paper presents improvements on a system for large-scale learning known as "parameter server". The parameter server is designed to perform reliable distributed machine learning in large-scale industrial systems (1000's of nodes). The architecture is based on a bipartite graph composed by "servers" and "workers". Workers compute gradients based on subsets of the training instances, while servers aggregate the workers' gradients, update the shared parameter vector and redistribute it to the workers for the next iteration. The architecture is based on asynchronous communication and allows trading-off convergence speed and accuracy through a flexible consistency model. The optimization problem is solved with a modified proximal gradient method, in which only blocks of coordinates are updated at a time. Results are shown in an ad-click prediction dataset with O(10^11) instances as well as features. Results are presented both in terms of convergence time of the algorithm and average time spent per worker. Both are roughly half of the values for the previous version of the parameter server (version called "B" in the paper). Roughly 1h convergence time using 1000 machines each with 16 cores and 192Gb RAM, 10Gb Ethernet connection (800 workers and 200 servers). Other jobs were concurrently run in the cluster. The authors claim it was not possible to compare against other algorithms since at the scale they are operating there is no other open-source solution. In the supplementary material they do compare their system with shotgun [7] and obtain faster convergence (4x) and similar value of the objective function at convergence. Since this is really a report of a working system, I have basically no technical comments to make. The numbers are impressive, the system works and there is no other open source system to compare against at this scale, so this is clearly a significant achievement. A nice materialization of a number of simple but important insights from the design of robust distributed systems into a machine learning pipeline (eg ideas like bounded delay, asynchronous push-pull, caching keys, etc.)Novelty claimed with respect to "existing approaches":- Solving large-scale problems (1000's nodes)- Solving non-convex and non-smooth composite objective functionsDoesn't [10] addresses these aspects (apart from non-smoothness, not sure?) If you refer only to open source versions, then make this clear.The aspect of the paper that I didn't like was the presentation.The authors compare the results of their so-called "third-generation parameter server" to two other systems called "system A" and "system B" which are not cited and are presumably the previous versions of their system. One aspect of the paper that makes assessment harder is that the contributions of the paper are not very clear in a case in which this clarification is quite important. This is precisely because there have been previous versions of the system that have presumably not been published (otherwise systems A and B would have been cited, at least anonymously) and the current submission conflates novelties from the latest version with aspects that originates in the earlier systems. In my view it would have been better if either the paper was written as introducing the parameter server from scratch, and illustrating its evolution up to this point (if the goal were to sell the entire package) or instead that the *precise* differences between the third generation and the previous ones were made really clear in the paper (if the goal were to sell the 3rd generation, and not the entire package). From the presentation I believe the goal was the latter. The table in page 7 helps, but I was expecting a much more detailed discrimination.Also, it's unfortunate that all the comparisons against other approaches (not the parameter server) are relegated to the supplementary material. I think this is a major weakness of the presentation.My suggestion is that the authors improve the presentation of the paper. I suggest they tell the story as introducing the parameter server and describing 3 stages of its evolution. Also the paper would read better if the English was polished (although to me it's acceptable as it is since it doesn't compromise understanding).