I have same issues. Masking output of softmax will sometimes comes to  nan / -inf.
I found out this is because the output of softmax is too small to float type.
The output of my NN is about -30. Sum of e^(-30) is nearly 0, which makes inf and nan.