Snap ML: A Hierarchical Framework for Machine Learning
We describe a new software framework for fast training of generalized linear models. The framework, named Snap Machine Learning (Snap ML), combines recent advances in machine learning systems and algorithms in a nested manner to reflect the hierarchical architecture of modern computing systems. We prove theoretically that such a hierarchical system can accelerate training in distributed environments where intra-node communication is cheaper than inter-node communication. Additionally, we provide a review of the implementation of Snap ML in terms of GPU acceleration, pipelining, communication patterns and software architecture, highlighting aspects that were critical for achieving high performance. We evaluate the performance of Snap ML in both single-node and multi-node environments, quantifying the benefit of the hierarchical scheme and the data streaming functionality, and comparing with other widely-used machine learning software frameworks. Finally, we present a logistic regression benchmark on the Criteo Terabyte Click Logs dataset and show that Snap ML achieves the same test loss an order of magnitude faster than any of the previously reported results, including those obtained using TensorFlow and scikit-learn.
The work at hand describes a new distributed implementation (Snap ML) for training generalized linear models for very large datasets. More precisely, the authors extend the popular CoCoA method [15] by providing a hierarchical version that is optimized for distributed computing environments. One of the key ingredients of this new version is the reduction of the overhead caused by the induced inter-node communication (see Section 2.1). In addition to a theoretical analysis of their new approach (Equation (2)), the authors also provide various implementation details including details related to an efficient local GPU solver per compute node (Section 3.1), to buffering techniques to reduce the overhead caused by memory transfers between host and device per compute node (Section 3.2), to an efficient exchange of information between compute nodes (Section 3.3), and to the overall software architecture. The experimental evaluation indicates some benefits of the new hierarchical scheme (Figure 5). It also demonstrates the benefits of the new framework for the Terabyte Click Logs.  The paper is well written and technically sound. I also like the theoretical analysis (Appendix B) of the inter-node communication pattern that is induced by parallelyzing the work per compute node (i.e., by having two loops of parallelism). It think the paper definitely meets the very high NIPS standards. My two main concerns are (1) lack of novelty and (2) the experimental evaluation:  (1) The framework extends the CoCoA framework. While being technically and theoretically sound, I find the practical merits of the approach quite limited. In particular, I am not blown away by the performance gains obtained via the new hierarchical framework. For instance, even given a slow network, I do not find the runtime reduction significant (e.g., 87 seconds instead for t_2=1 compared to 67 seconds for t_2 \approx 10, see Figure 5). I have the feeling that a similar performance gain could be achieved by compressing a shared vector v prior to sending it over the network.  (2) The approach is clearly of practical importance. However, I was not fully convinced by the experimental evaluation. For instance, in Figure 4, I find the performance comparison between Snap ML, Scikit-Learn, and TensorFlow not entirely fair: Scikit-Learn only makes use of a CPU instead of a V100 GPU. Further, TensorFlow reads the data in batches from disk (as stated by the authors). Thus, Snap ML is faster than Scikit-Learn simply due to the GPU and faster than TensorFlow due to the slow transfer between disk and memory device. Can one improve the performance of TensorFlow by increasing the batch size or by buffering/caching? Also, while I find the results in Figure 7 impressive, I am missing one important baseline: Snap ML with t_2=1, that is, a Spark+GPU version of the previous CoCoA framework (without the additional hierarchical additions). What is the performance of this variant? Do the modifications proposed in the work yield significant better results compared to a rather direct distributed application of the existing framework (i.e., a direct Spark/GPU implementation of the CoCoA implementation)?  Based on the authors' comments, I have decided to increase my score a bit. Overall, it is a decent contribution. I nevertheless think that the experimental evaluation could be improved.  Minor comments: - Figure 5: hierarchical