Are you concerned about data loading speed due to the files being on disk rather than in-memory? I believe that if there is enough main memory to cache the files, the filesystem will do this by default so any additional savings from caching the loaded tensor may be minimal. Furthermore, it seems this would require turning off shuffling (which you have correctly done), which may hurt model accuracy.