I encountered the same problem with PyTorch 1.4.0. At first I thought it couldn’t converge. But after a [very] * 100 long time, it started to converge [very] * 100 slowly.
And then I turned to DistributedDataParallel, and it worked. There may be a bug in LSTM + DP, I guess.
Check out how to use DDP here: https://github.com/pytorch/examples/blob/master/imagenet/main.py 8