There might some issue with how you use RNN module. Make sure that the final classification is from the last hidden state. And maybe tune the parameter a bit.
BTW, I updated the above reference with seq2seq language models.