Does your code run if you do not use self.lstm1.flatten_parameters()? If so, it could be because you are using DataParallel, can you try using DistributedDataParallel instead? It also depends on the pytorch version that you are using, the fix above seems to work up to pytorch 0.3.0. If you are using pytorch 0.3.1 or 0.4, there is an issue about it on github here: https://github.com/pytorch/pytorch/issues/7092 281