First, can you make sure there are no NANs in your data? Sometimes this can accidentally happen during data augmentation.
I.e., just before the forward pass, please print:
torch.sum(torch.isnan(data))