Polyaxon – An open source platform for reproducible machine learning at scale
I&#x27;ve been starting to use this in spaCy, so I&#x27;m glad to see it posted here! It&#x27;s still a young project, but Mourad has been very dedicated to it, and I think it&#x27;s already at the point where it&#x27;s useful. I hope more people can contribute. Here&#x27;s a quick review.<p>Most people doing machine learning at the moment are using a pretty bad workflow. It&#x27;s difficult to avoid the trap people refer to as &quot;grad student descent&quot;: endless tinkering, where you run two or three jobs, monitor the results, and then kick off another one. You don&#x27;t really have a hypothesis in this cycle, so you don&#x27;t know when to stop. At the end of the process you&#x27;ve generally gained intuition and insight, but nothing you can reliably pass on.<p>The solution to this trap is to commit to a matrix of results you&#x27;re going to collect, program up the experiments, and let them run. Once you have the proper comparison, you can then decide what to do next.<p>Most university research groups get a grant to buy some hardware, create a cluster, and then schedule jobs on the machines using SLURM or HTCondor. These technologies are mature, but they leave individual researchers with a lot to do. You can schedule your jobs, but you have to write the experiment management yourself. My hunch is there are maybe 5-50 companies in the world with internal systems significantly more sophisticated than this.<p>Polyaxon brings the shiny new &quot;cloud native&quot; workflow to the problem. It runs on Kubernetes, which is much easier to use, especially with heterogenous hardware. I think just switching to Kubernetes and containers would be helpful to a lot of teams. On top of the cluster solution, Polyaxon brings a nice experiment management layer, with hyper-parameter search. It also manages the containerisation for you, so that the researcher doesn&#x27;t need to interact with Docker directly.<p>There are still a number of things that are under-developed. The most noticeable are dataset management and artifact export. You currently have to do this yourself, e.g. by adding persistent disks to the cluster. I use GCSFuse to mount GCS (Google S3) buckets as directories, which works pretty well in the meantime. There are also a few defaults that could use refinement. If large clusters are being created, the management services are currently a bit under-resourced. Finally, there are a few more minor rough spots. For instance, the web app is currently a little unpolished. It&#x27;s a Django app, so everything takes two or three more clicks and refreshes than you&#x27;d ideally want. A more AJAXy front-end would be nice.<p>There are several commercial competitors. There&#x27;s a pretty obvious analogy between the use-case here and the CRM. Companies hope to own the &quot;system of record&quot;, and be the shared space where ML teams collaborate. DominoDataLab.com , Neptune.ml , Cloudera.com , Datascience.com and others all have slightly different takes on this problem, to different degrees. Many of the above are built around a Jupyter Notebooks-based experience, and are targeted more towards workflows where the primary outputs are insights and reports rather than product development.<p>I think an open-source framework is valuable for a few reasons. We should be reluctant to buy in to commercial platforms for precisely the reasons vendors are so interested in this space. Lock-in can hurt here, and if the computation is scheduled via the vendor, they get the chance to tax you a % of your compute spending. Given how expensive GPU experiments can be, that&#x27;s a big ongoing cost to sign up for.<p>I also think uploading all your training data to someone else&#x27;s system is a bad idea, that your data sharing agreements often won&#x27;t permit.<p>Finally, it&#x27;s nice to have a local Kubernetes cluster for other reasons. Kubernetes is basically an OS. Polyaxon is an app that runs on that OS. This is nice: you can develop other apps to work with it as well. In contrast, if you rent a service, the easiest way to meet your next requirements will be with further services. As soon as you hit custom requirements, costs and risks rise rapidly. The in-house approach signs you on to a better future --- it&#x27;s a step in the right direction. The commercial services may or may not be easier today, but at some point you&#x27;ll want to switch out of them. It may still be worth using them today --- but the long-term perspective is at least a point in Polyaxon&#x27;s favour.