Hi Patrick, sorry for the revival of an old post.
I am using a huge dataset (8 GB) of numpy arrays.
I have tried an lmdb dataset approach, which involved conversion of the arrays into .jpg formats, I haven’t been able to reproduce my previous results on arrays.
The solutions here are ideal, but would you say it is a bad idea to load .npy arrays every minibatch? It seems people don’t tend to do this!