On Model Parallelization and Scheduling Strategies for Distributed Machine Learning
Distributed machine learning has typically been approached from a data parallel perspective, where big data are partitioned to multiple workers and an algorithm is executed concurrently over different data subsets under various synchronization schemes to ensure speed-up and/or correctness. A sibling problem that has received relatively less attention is how to ensure efficient and correct model parallel execution of ML algorithms, where parameters of an ML program are partitioned to different workers and undergone concurrent iterative updates. We argue that model and data parallelisms impose rather different challenges for system design, algorithmic adjustment, and theoretical analysis. In this paper, we develop a system for model-parallelism, STRADS, that provides a programming abstraction for scheduling parameter updates by discovering and leveraging changing structural properties of ML programs. STRADS enables a flexible tradeoff between scheduling efficiency and fidelity to intrinsic dependencies within the models, and improves memory efficiency of distributed ML. We demonstrate the efficacy of model-parallel algorithms implemented on STRADS versus popular implementations for topic modeling, matrix factorization, and Lasso.
The authors proposed a framework of primitives for model parallelism, in order to address the problem that a single machine may be inadequate to fit a big model in memory.This seems to be an interesting paper, well written and organized, albeit some small typos are found (e.g., q_idx -> idx in Figure 4, 1/L -> 1/P in line 140).Toward the significance of this paper, I have the following concerns:1. I agree that for models such as deep neural networks (e.g. CNN), more parameters often leads to better performance, as demonstrated empirically by [5] (using the paper's bib reference number), which justifies the need of more efficient model parallelism framework for bigger models. However, for models like topic models (LDA) and matrix factorization (MF), I am skeptical toward such need. For example, would LDA with more than tens of thousands of topics, or MF with rank larger than a few thousands, lead to better model performance, compared against their counterparts with moderate number of parameters? The work could be more interesting if the authors address this issue. Besides, can the proposed framework be applied to models that already proved to enjoy more parameters, such as deep neural networks? If yes, how would it perform when compared to the one proposed in [5]? In short, I think more compelling use cases for the proposed framework would greatly enhance the potential impact of this work on the NIPS community.2. The experiments.1) Some important specifications for the experiments are missing. On the evaluation metrics for LDA and MF, how was the convergence being measured? Take LDA, was the log-likelihood calculated on training documents or testing (unseen) documents? One would imagine LDA with huge number of topics can easily over-fit, if no special regularization or sparsity-inducing priors being adopted, and such convergence could be meaningless. And how was the memcached (which is one of the keys for the scalability for YahooLDA) configured? And for the proposed framework, how were the summary statistics D and B stored and distributed to the work nodes, as their size grow proportionally to the number of parameters? These are just some key factors that may need to be clarified, as they may contribute to the difference observed in Figure 8 and 9.2) The comparison between STRADS and Graphlab in the MF experiment is somehow unfair. Because STRADS used the cyclic coordinate descent (CCD) as its optimization algorithm, whereas Graphlab used alternating least square (ALS). As it has been demonstrated in [21], CCD converges faster than ALS and is also more scalable. As a result, it would be more interesting to compare STRADS against Graphlab with the *same* optimization algorithm. Furthermore, the method proposed in [9] is related, and would also be an interesting baseline for comparison. With such additional experimental results, one may gain further insight into the STRADS framework.