Spark, Distributed Systems, Machine Learning, Hadoop, Scalability.
Given a task across huge amount of data, distributed across numerous nodes, a lot of data transfer has to happen and processing needs to be distributed. Let’s look into this in detail. Map-Reduce Paradigm Consider the task to find word frequency in a large distributed file of 900 GB. HDFS will