Could you check the tensors for invalid values before and after applying the transform and run a few epochs? Based on your description it seems that this issue is not reproducible, but is visible randomly?