Modifying tensors in-place can cause issues with backprop. Here is my solution, since this is still ranked highly on google:
safe_tensor = torch.where(torch.isnan(my_tensor), torch.zeros_like(my_tensor), my_tensor)