Thanks for the info.
The strange loss behavior might be also related to your sampling strategy or the input in general.
Could you check all inputs for NaNs or Infs in your training loop or beforehand?
torch.isnan or torch.isinf might be helpful here.