I’ll run that model and see. Any specific reason why you applied two activation functions to a single layer? The last layer has a SELU and a log softmax together without a layer in between those two activations. Also, you are applying logsoftmax to a layer which contains previous timesteps stacked horizontally. For instance, consider the sequence [1,2,3,4]. Now if I put in 1,2,3, we train the RNN to output 2,3,4. But in your case, since x=x.contiguous().view(self.batch,-1) , you are stacking the timesteps which contain 2,3,4 horizontally and then applying log softmax. So the RNN will be forced to chose just one of the character, it will try to learn 4 since that is the label in the dataset. I on the other hand transform it in this way,
x=x.contiguous().view(batch_size*seqlen,vocab_size).
This way, I stack 2,3,4 vertically and if I apply softmax, I get three different outputs across three different time steps. And then I can train the RNN to output 2,3,4 in that order depending on the input.