Spark, Distributed Systems, Machine Learning, Hadoop, Scalability.
images from ECR, we set container environment variable YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG. However, we need to generate config.json and put in HDFS so that it can be used by jobs running on the cluster. To do this, login to one of the core nodes and execute below commands. ssh -i