Dear KFrank,
Thank you very much for your reply. It makes perfect sense what you said. Now, I implemented my model as you said and it is fine. However, now the loss becomes nan after awhile. I ran torch anomaly detection and the problem is caught there. It tells me
RuntimeError: Function ‘LogBackward’ returned nan values in its 0th output.
and the last called line is the following: (Basically where I do log(1-s))
local_distribution = torch.log(1-st) + F.log_softmax(attention_weights.squeeze(dim=2), dim=1)
I thought about adding a small perturbation to both logs. Somethinglike:
perturbation = 1e-6
w1 = log(s + perturbation)
w2 = log(1 - s + perturbation)
Do you think that it makes sense or any do you have any other suggestions ?