I think your question is oriented towards X being a business problem.<p>Netflix has users (say 100M) who have been liking some movies (say 100k). Say The question is: for every user, find movies he&#x2F;she would like but have not seen yet.<p>The dataset in question is large, and you have to answer this question with data regarding every user-movie pair (that would be 1e13 pairs). A problem of this size needs to be distributed across a cluster.<p>Spark lets you express computations across this cluster, letting you explore the problem. Spark also provides you with a quite rich Machine Learning toolset [1]. Among which is ALS-WR [2], which was developped specifically for a competition organised by Netflix and got great results [3].<p>[1] <a href="http:&#x2F;&#x2F;spark.apache.org&#x2F;docs&#x2F;latest&#x2F;mllib-guide.html" rel="nofollow">http:&#x2F;&#x2F;spark.apache.org&#x2F;docs&#x2F;latest&#x2F;mllib-guide.html</a>
[2] <a href="http:&#x2F;&#x2F;spark.apache.org&#x2F;docs&#x2F;latest&#x2F;mllib-collaborative-filtering.html#collaborative-filtering" rel="nofollow">http:&#x2F;&#x2F;spark.apache.org&#x2F;docs&#x2F;latest&#x2F;mllib-collaborative-filt...</a>
[3] <a href="http:&#x2F;&#x2F;www.grappa.univ-lille3.fr&#x2F;~mary&#x2F;cours&#x2F;stats&#x2F;centrale&#x2F;reco&#x2F;paper&#x2F;MatrixFactorizationALS.pdf" rel="nofollow">http:&#x2F;&#x2F;www.grappa.univ-lille3.fr&#x2F;~mary&#x2F;cours&#x2F;stats&#x2F;centrale&#x2F;...</a>