I solved the problem by deleting the init_hidden function.
I used to call the init_hidden every time before forward function just like the tutorial on the official website and initialized the hidden variables with (batch_size/num_gpu). It might cause error in dataparallel.
By the way, the warning  “UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greately increasing memory usage. To compact weights again call flatten_parameters().” still exists, but everything seems OK.