Hi Berkay!



 berkay_berabi:

RuntimeError: Function ‘LogBackward’ returned nan values in its 0th output.
and the last called line is the following: (Basically where I do log(1-s))
local_distribution = torch.log(1-st) + F.log_softmax(attention_weights.squeeze(dim=2), dim=1)


Have you verified that the divergence occurs in log (1 - s), and
not, perhaps in a log() embedded in log_softmax()?  Can you
check that s has become greater than (or equal to or close to) 1.0?


perturbation = 1e-6
w1 = log(s + perturbation)
w2 = log(1 - s + perturbation)


This would likely cure the immediate problem, but I would probably
use torch.clamp(), rather than adding perturbation.


Do you think that it makes sense or any do you have any other suggestions ?
I have a learnable scalar s (in range [0,1])


Is there anything in your training or the structure of your problem
that “knows” that s is supposed to be a probability, and keeps it
between 0 and 1?  Your training might happily push s outside its
valid range, leading to nans in log().
I would would suggest training the logit of s.  (It will run from -inf to
inf.)  Let’s call it t.  Then replace log (s) and log (1 - s) with:
torch.nn.logsigmoid (t)
torch.nn.logsigmoid (1.0 - t)

(where we’re using pytorch’s logsigmoid() to avoid the potential
numerical instability of calling log (sigmoid (t)) in two steps).
Best.
K. Frank