Building An Aws Pipeline, Data Science, Big Data, AWS, Towards Data Science.
executable inside the cluster (as opposed to from our home, non AWS machine), we ask the Spark framework to use 20GB of memory when running our scripts and its other steps (actually our script gets a little less because of how memory allocation works between Spark and YARN) with the commands