Building An Aws Pipeline, Data Science, Big Data, AWS, Towards Data Science.
Runner",Jar="command-runner.jar",Args=["spark-submit","--deploy-mode=cluster","--executor-memory","20G","s3://workflow-scripts/process_data.py, s3://data-files/AAPL.csv"] — This one is pretty dense so let’s break it up even more. First we will use the command-runner.jar Jar file to run our python