title: How do I install my code and dependencies in an AWS Spark cluster?
body: <p>I can create a Spark cluster on AWS as described <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-spark-launch.html">here</a>.</p>
  
  <p>However, my own Python code and pip libraries need to run on the master and workers. This is a <em>lot</em> of code, and the pip installation process compiles some native libraries as well, so I can't simply have Spark distribute this code at runtime using techniques such as <a href="https://issues.apache.org/jira/browse/SPARK-5929">registering a pip requirements file with spark_context</a> or the <a href="https://spark.apache.org/docs/1.2.0/submitting-applications.html">--py-files argument of spark-submit</a>.</p>
  
  <p>Of course I could run a bash script right after  running <code>aws emr create-cluster</code>, but I wonder if there is a  more automatic way, so that I can  avoid the maintenance of a big bash script for installation.</p>
  
  <p>So, what is the best way to  set up clusters to include my code and dependencies?</p>