As far as I know the nan values will be propagated by mathematical operations.
ReLU seems to zero out the nans:
x = torch.randn(1, 3, 10, 10)
x[0, 0, 0, 0] = torch.log(torch.tensor([-1.]))

m = nn.Conv2d(3, 6, 3, 1, 1)
output = m(x)
print(output[0, 0, 0:3, 0:3])
> tensor([[    nan,     nan,  0.5057],
        [    nan,     nan, -0.4137],
        [-0.8564,  0.1068, -1.0790]])

r = nn.ReLU()
output = r(output)
print(output[0, 0, 0:3, 0:3])
> tensor([[ 0.0000,  0.0000,  0.5057],
        [ 0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.1068,  0.0000]])

I would try to look for the origin of the nan values, because it seems to be wrong in most cases.