In a word, torch.load() is taking about 30s-1min to load a 300Mb file. I can’t directly say this is too slow, but for older datasets of mine it’s almost instantly done. Here’s the way I’m doing it:
I have a large set of data in the form of csv array of size about 500Mb. But I found this is very slow to load using np.genfromtxt, so I load the files, convert them to torch tensors, divide it into 150 subsets in some partition and use torch.save() to save them. What might went wrong during the process?