You could clamp these invalid values or just use logsigmoid as seen here:
x = torch.tensor([-100., 100])
print(torch.log(torch.sigmoid(x)))
> tensor([-inf, 0.])
print(torch.nn.functional.logsigmoid(x))
> tensor([-100.,   -0.])