Solved by mrshenli in post #2 
              
              
                It depends on what you expected reset_hidden_state to achieve. Below is what happens in EVERY forward pass when you use DataParallel. 

split input data
replicate model to all devices
feed input data splits to all model replicas
gather outputs from all replicas
done with forward

After the forward …