Problem:
Hello everyone, I’m working on the code of transfer_learning_tutorial by switching my dataset to do the finetuning on Resnet18.
I’ve encountered a situation where the batch loss turns quickly (at 2nd ot 3rd batch) to infinitive and then nan  if I shuffle the training data in my data_loader.
Solution tried
However, if i give the option shuffle=False in the dataloader, the problem is still there but appears much later and randomly(the batch where the loss turns to nan is different each time I run the code) . But sometimes one epoch could be completely finished so that I supposed it might not be the problem of my dataset.
I use a initial learning rate of 1e-5 with learning rate decay by 7 epochs.
##Description of code
My images and labels are saved in a large hdf5 file (uint8 for images and float32 for labels). I rewrite the dataloader and apply some preprocessing on my data. I use the MSELoss as the loss function and all other parts are quite similar to the tutorial.

    class MyDataset(torch.utils.data.Dataset):
        def __init__(self, image_h5, label_h5):
             assert image_h5.shape[0] == label_h5.shape[0]
             self.image_h5 = image_h5
             self.label_h5 = label_h5

        def __getitem__(self, index):
             return torch.from_numpy(((self.image_h5[index]-MEAN_MATRIX)/255).astype(np.float32)), torch.from_numpy(self.label_h5[index])

        def __len__(self):
             return self.image_h5.shape[0]
train_set = MyDataset(train_file['image_set'],train_file['label_set'])
test_set = MyDataset(test_file['image_set'],test_file['label_set'])
dset_loaders ={'train':torch.utils.data.DataLoader(train_set, batch_size=opt.batch_size,
                                               shuffle=True, num_workers=opt.threads),
				'val':torch.utils.data.DataLoader(test_set, batch_size=opt.batch_size,
                                               shuffle=False, num_workers=opt.threads)}


I would like to know the probable reason for this problem and if I’ve rewritten correctly my dataloader.
Thank you very much