An update. Instead of sigmoid, I used nn.LogSoftmax. Sometimes it trains and sometimes the outputs are nan. But I get this warning:
UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.

What are your thoughts on this?