I finally got a close result to the Caffe version after I clarified some differences between Caffe and Pytorch:

SGD implementation.
dropout is not applied if there is only one RNN layer.

and also I found the data preprocessing of my pytorch version is slightly different from the Caffe version.
After I solved these problems, I got a comparable result.
Hope this helps!