Pyspark, Mongodb, AWS, Machine Learning.
applications it will be distributed to 8 threads file_paths = sc.parallelize(h5_file_paths, 8) # Read and convert to python dict in parallel songs_rdd = file_paths.map(lambda x: h5todict(x)) songs = songs_rdd.collect() Here we also collect songs as a python list in order to create our MongoDB