title: spark in yarn-cluser 'sc' not defined
body: <p>I am using spark 1.3.1.</p>
  
  <p>Do I have to declare sc when spark run in yarn-cluster mode?  I have no problem running the same python program in spark python shell.</p>
  
  <p>This is how I submit the job :</p>
  
  <pre><code>/bin/spark-submit --master yarn-cluster test.py --conf conf/spark-defaults.conf
  </code></pre>
  
  <p>where in spark-defaults I did declare where the <code>spark.yarn.jar</code> is, also check permission on where <code>spark.yarn.jar</code> is and <code>/user/admin</code>, the spark user, to make there is read-write-execute for all.</p>
  
  <p>In my <code>test.py</code> program, I have <code>from pyspark.sql import SQLContext</code> and the first line is </p>
  
  <pre><code>sqlctx=SQLContext(sc)
  </code></pre>
  
  <p>and the error is </p>
  
  <pre><code>NameError: name 'sc' is not defined
  </code></pre>
  
  <p>on that line.</p>
  
  <p>Any idea?</p>