Well, if you set num_workers = 0, you will take a performance hit. it seems that the issue is related to multiprocessing then.
Could you change to use torch.load and torch.save for your tensors? Pickling tensors with default pickle module is known to be very slow in some cases.