Spark, Spark Core, Rdd, Spark Api.
sc.wholeTextFiles(name, partitions). This method lets you read a directory containing multiple small text files, and returns each of them as (filename, content) pairs. This is in contrast with textFile, which would return one record per line in each file. Some notes on reading files with Spark: If