Hmm, ill try h5 format
I used concatdataset because i thought it would manage loading different pieces of data across different files without having to load them into memory all at once, but looks like that is what it is doing, exactly