thanks for the comment, as you mentioned, I put the code in forward():
...
   x_rnn_input = x.view(batch_size,-1,rnn_input_size)  
    with torch.backends.cudnn.flags(enabled=False): 
        output_rnn, hidden_rnn = self.rnn_layer(x_rnn_input
...

but the same result is given.  (yes, the dropout = 0 is set as well)