Effective Parallelisation for Machine Learning
We present a novel parallelisation scheme that simplifies the adaptation of learning algorithms to growing amounts of data as well as growing needs for accurate and confident predictions in critical applications. In contrast to other parallelisation techniques, it can be applied to a broad class of learning algorithms without further mathematical derivations and without writing dedicated code, while at the same time maintaining theoretical performance guarantees. Moreover, our parallelisation scheme is able to reduce the runtime of many learning algorithms to polylogarithmic time on quasi-polynomially many processing units. This is a significant step towards a general answer to an open question on the efficient parallelisation of machine learning algorithms in the sense of Nick's Class (NC). The cost of this parallelisation is in the form of a larger sample complexity. Our empirical study confirms the potential of our parallelisation scheme with fixed numbers of processors and instances in realistic application scenarios.
The authors have presented a parallelization algorithm for aggregating weak learners based on Radon partitioning. They present theoretical analysis to motivate the algorithm along with empirical results to support the theory. The theoretical analysis is interesting, and the empirical results demonstrate training time and/or AUC improvements over multiple baseline algorithms, on multiple datasets. The authors also preemptively and convincingly address several questions/concerns in the Evaluation and Discussion sections. I recommend acceptance.
Specific Notes
--------------
- Line 51: "the the" -> "the"
- Line 79: Is this the same size math font as elsewhere? It seems small and kind of hard to read.
- Figures 1-3: The text really needs to be bigger in the figure images.
- Figure 3: This one may be harder to distinguish if printed in black and white.
- Lines 318-320: "the Radon machine outperforms the averaging baseline..." To clarify, is this based on an average of paired comparisons?
- Line 324: A pet peeve of mine is the use of the word significant in papers when not referring to statistical significance results.
- Lines 340-341: Missing some spaces - "machineachieved", and "machineis."
- Lines 341-349: The difference in performance with the AverageAtTheEnd method doesn't seem terribly meaningful. Why wasn't this result included somewhere in Figure 1? Can you advocate more for your method here?
- Line 377: "Long and servedio [22]" The use of author names here seems inconsistent with citations elsewhere (even immediately following).