Thanks for improving your code, there were two small places I needed to fix as they didn’t go through well on my machine. one in linear_nce.py:
unigram_prob = torch.Tensor(unigram_prob)+1e-10
self.unigram_prob = Variable(unigram_prob, requires_grad=False).cuda()
and the second in ice_loss.py:
loss = F.binary_cross_entropy_with_logits(logits, nce_target)
The problem this time was that it did it seemed to get stuck at some point and it took it forever to complete a round (I played with a toy example that previously produced the 'Nan’s, and this time it never riches the point of printing the ppl as it never finished an epoch). I didn’t debug further to investigate the problem, but I wonder whether it runs on your end?