CYCLADES: Conflict-free Asynchronous Machine Learning
We present CYCLADES, a general framework for parallelizing stochastic optimization algorithms in a shared memory setting. CYCLADES is asynchronous during model updates, and requires no memory locking mechanisms, similar to HOG-WILD!-type algorithms. Unlike HOGWILD!, CYCLADES introduces no conflicts during parallel execution, and offers a black-box analysis for provable speedups across a large family of algorithms. Due to its inherent cache locality and conflictfree nature, our multi-core implementation of CYCLADES consistently outperforms HOGWILD!-type algorithms on sufficiently sparse datasets, leading to up to 40% speedup gains compared to HOGWILD!, and up to 5⇥ gains over asynchronous implementations of variance reduction algorithms.
In this paper, the authors propose an enhancement to the HogWild! Framework, which is claimed to maintain serial equivalence (i.e., produces the same outcome as the serial algorithm). Experimental results demonstrate that the proposed method outperforms HogWild! on a few tasks. 
1) I am afraid that the paper contains over-claims. The Cyclades method actually only conditionally achieves the serial equivalence according to Theorem 1. The conflict degree is highly related to the sparsity of the data, and the condition of Theorem 1 cannot be satisfied in many practical scenarios. For example, for deep learning and other dense applications, the allocation algorithm cannot resolve conflicts at all. Even for sparse applications, when the data block is large, or the delay among different local machines is large, the serial equivalence cannot be achieved either. 2) As for the technical contribution of the paper, it seems not that significant over Hogwild!. To some extent, Cyclades adds a data partitioning step to Hogwild, and this data partitioning is done by leveraging existing bipartite allocation algorithms. 3) The authors made some analysis on the speed up achieved by Cyclades. However, speed up is only a system measure, and from the machine learning perspective, we care more about the balance between accuracy and efficiency. The speed up measure is defined with respect to \epsilon objective, which is a little tricky. \epsilon is defines as the smallest error achievable by parallel algorithm (but not by the serial algorithm) – this is unfair to the serial algorithm. According to my experience, in some cases, the parallel algorithm cannot achieve the same smallest error as the serial algorithms do. So, if we define \epsilon as the smallest error by the series algorithm instead, the parallel algorithm will have no speed-up at all. 4) As for the baseline in the experiment, the authors only compare with HogWild!. Given that there have many so many parallelization mechanisms proposed for distributed machine learning these years, only comparing with Hogwild! cannot sufficiently demonstrate the value of the proposed approach. It is necessary to compare with more and stronger baselines.  In general, I am not convinced by the rebuttal. According to the authors, serial equivalence is theoretically guranteed, but to me it is guaranteed in a very trivial way. When the data are dense, in order to gurantee the serial equivalence, one may have to use one single core which will make no sense of parallelization. Only if the data is sparse, the partition algorithm can take effect. Furthermore, many parallelization algorithms design for distributed setting can be easily adapted to the shared-memory setting, therefore only comparing with HogWild! seems not convincing enough to me.