Are you sure there’s an actual underflow problem there and not simply that the printout only shows four decimals?
Try printing torch.exp(z) - 1 instead: tensor([1.1102e-15, 1.9984e-15, 3.1086e-15], dtype=torch.float64)
That looks right to me (with rounding errors due to floating point et c of course), since \lim_{x \to 0} e^x = x and the log_softmax for so small differences shouldn’t be visible.
Unless I’m completely misunderstanding something, in which case I’m looking forward to learning something new when you respond