We use it for two things:<p>* distributed machine learning tasks using their built-in algorithms (although note that some of them, e.g. LDA, just fall over with not-even-that-big datasets)<p>* as a general fabric for doing parallel processing, like crunching terabytes of JSON logs into Parquet files, doing random transformations of the Common Crawl<p>As a developer, it&#x27;s really convenient to spin up ~200 cores on AWS spot instances for ~$2&#x2F;hr and get fast feedback as I iterate on an idea.