I didn’t think about sharding the dataset but great idea!
The file path are naturally organized based on their hash so I could try to zip them based on the first 3 letters.
Didn’t know about JSON Lines, I’ll give it a try.