But in a larger graphical models like CTC or HMMs, this is introducing nans in my backpropagation, What could be the best possible way to avoid this behavior; as I want my model to be training on the non-infinity parameters in the different locations what I mean is that consider a matrix;
x = torch.tensor([[ 1.0, -float("inf")], [-float("inf"), -float("inf")]])
>>> x.shape
torch.Size([2, 2])
>>> x
tensor([[1., -inf],
        [-inf, -inf]])
>>> x = nn.Parameter(x)
>>> y = torch.logsumexp(x, dim=1)
>>> y
tensor([1., -inf], grad_fn=<LogsumexpBackward>)
>>> z = torch.sum(y, dim=0)
>>> z.backward()
>>> x.grad
tensor([[1., 0.],
        [nan, nan]])

Take this example for say; how can I make changes that instead of nan I get zero gradients at x.grad?