A Comparison of Distributed Machine Learning Platforms
I would like to encourage anyone to read these kinds of &quot;evaluations&quot; with a bit of criticism.<p>When doing evaluations there are a lot of nuances.<p>I will comment on a real world example
from our own experience.<p>Disclaimer: I am biased running a company 
implementing my own platform.
I will try to stick to topics that folks should &quot;look for&quot; when evaluating these things rather than inviting a debate on &quot;my tool vs yours&quot;<p>For deep learning on spark specifically, there are 2 workloads, training and inference.<p>Your data pipeline here matters a lot. A job that pulls data from hive will <i>surprise</i> be bottlenecked by hive.
This by extension means the kind of data and small things like the number of records you fetch or cache on say: hdfs matters a TON.<p>On our platform, we implement 2 versions of distributed communications: spark driver and parameter server.  The parameter server is faster. The latter is what mxnet and tensorflow do as well. Tensorflow uses grpc underneath.
mxnet writes its own.<p>While both are data flow graphs, the latter is optimized for linear algebra and I would like to say has very different requirements (for example gpus which are another level of complexity)<p>When using these parameter servers, you should look at what guarantees these things provide. For example, do you need consistent latency? Fault tolerance? At what scale?<p>Something as simple as&quot;do these parameter servers do p2p on each node?&quot;<p>With spark, it&#x27;s not well designed for that. A lot of workers are meant to be stateless.<p>With spark another gotcha (especially on the linear algebra part) is on heap memory. When you are using say: pyspark for example, how you handle data movement matters a lot, and again <i>surprise</i> ends up being your bottleneck.<p>We use pyjnius instead in our python interface for linear algebra:
<a href="https:&#x2F;&#x2F;github.com&#x2F;deeplearning4j&#x2F;jumpy" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;deeplearning4j&#x2F;jumpy</a><p>We implemented this because we liked the idea of being able to pass numpy pointers straight to the JVM rather than copying data. I believe pytorch and co do this as well (which are just straight C&#x2F;C++). That is the right way to do it<p>The default is py4j (which overall is more versatile but not meant for DL)<p>Don&#x27;t even get me started on a normal etl pipeline vs &quot;images&quot;, which is typically what you see.<p>Another big complexity with say: spark and even gpus is gpu usage. When using mesos, you can only reserve a whole gpu, not fractions.<p>Another thing is worker setup. For deep learning specifically, What BLAS did they use? How was it compiled?
Was it optimized for the right cpu architecture?<p>How does the   the workers communicate with the gpu, if any?<p>If you&#x27;re using a JVM, how do they handle the FFI?<p>How are workers configured for cpu&#x2F;cpu? 
Things like OMP_NUM_THREADS on each worker matter a lot.<p>If you&#x27;re using spark, did you configure k small workers or 1 big one? How did that interact with your multi threaded linear algebra?<p>Hopefully this gives people an idea of how complicated these things get.