Pyspark, Mongodb, AWS, Machine Learning.
to do this conversion in parallel using pyspark. The basic idea is to collect all h5 filepahts in a python list and create an RDD out of this list. Then we can easily do the conversion we want using map function on this RDD. # 8 is number of partitions to distribute the data # For local