title: How to fully utilize all Spark nodes in cluster?
body: <p>I have launched a 10 node cluster with the ec2-script in standalone mode for Spark. I am accessing data in s3 buckets from within the PySpark shell but when I perform transormations on the RDD, only one node is ever used. For example the below will read in data from the CommonCorpus:</p>
  
  <pre><code>bucket = ("s3n://@aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2014-23/"
            "/segments/1404776400583.60/warc/CC-MAIN-20140707234000-00000-ip-10"
            "-180-212-248.ec2.internal.warc.gz")
  
  data = sc.textFile(bucket)
  data.count()
  </code></pre>
  
  <p>When I run this, only one of my 10 slaves processes the data. I know this because only one slave (213) has any logs of the activity when viewed from the Spark web console. When I view the the activity in Ganglia, this same node (213) is the only slave with a spike in mem usage when the activity was run. <img src="https://i.stack.imgur.com/gMhCa.png" alt="enter image description here"></p>
  
  <p>Furthermore I have the exact same performance when I run the same script with an ec2 cluster of only one slave. I am using Spark 1.1.0 and any help or advice is greatly appreciated.</p>