Spark, Distributed Systems, Machine Learning, Hadoop, Scalability.
Distributed Dataset (RDD), a fault tolerant immutable collection of distributed datasets stored in main memory. On top of RDD, DataFrame API is designed to abstract away its complexity and ease doing Machine Learning on Spark. RDDs support two types of operations: Transformations: to create a new