Spark, Scala, Java, Big Data, Data Science.
run all the tasks, driver memory is also set to 8GB using the configuration spark.driver.memory. All of this is not required while running the job on a cluster as master is provided with the spark-submit command, or it gets picked up from the Spark cluster the job is submitted to, if the cluster is