title: PySpark : Setting Executors/Cores and Memory Local Machine
body: <p>So I looked at a bunch of posts on Pyspark, Jupyter and setting memory/cores/executors (and the associated memory).</p>
  <p>But I appear to be stuck -</p>
  <p>Question 1: <strong>I dont see my machine utilizing either the cores or the memory.</strong> Why?   Can I do some adjustments to the excutors/cores/memory to optimize speed of reading the file?
  Question 2: Also is there any way for me to see a progress bar showing how much of the file ahs been imported (spark-monitor doesnt seem to do it).</p>
  <p>I am importing a 33.5gb file into pyspark.</p>
  <p>Machine has 112 gb or RAM
  8 Cores/16 virtual cores.</p>
  <pre><code>from pyspark.sql import SparkSession
  </code></pre>
  <pre><code>spark = SparkSession \
      .builder \
      .appName(&quot;Summaries&quot;) \
      .config(&quot;spark.some.config.option&quot;, &quot;some-value&quot;) \
      .getOrCreate()
  conf = spark.sparkContext._conf.setAll([('spark.executor.memory', '4g'), 
  
  ('spark.app.name', 'Spark Updated Conf'), 
  ('spark.driver.cores', '4'), ('spark.executor.cores', '16'), 
  ('spark.driver.memory','90g')])
  spark.sparkContext.stop()
  spark = SparkSession.builder.config(conf=conf).getOrCreate()
  df = spark.read.json(&quot;../Data/inasnelylargefile.json.gz&quot;)
  
  </code></pre>
  <p>I assume that the pyspark is doing its magic even while reading a file (so I should see heavy core/memory utilization). But I am not seeing it.Help!</p>
  <p>Update: Tested with a smaller zip file (89 MB)</p>
  <p>Pyspark take 72 seconds
  Pandas takes 10.6 seconds
  Code used :</p>
  <pre><code>start = time.time()
  df = spark.read.json(&quot;../Data/small.json.gz&quot;)
  end = time.time()
  print(end - start)
  
  start = time.time()
  df = pa.read_json('../Data/small.json.gz',compression='gzip', lines = True)
  end = time.time()
  print(end - start)
  </code></pre>