I have ran a few models using DataParallel, just by simply passing the model to nn.DataParallel() and I have had a range of strange things that happen sometimes.  What I see commonly ,is my graphs (matplotlib, etc) will look different, like WAY different, which makes me think there is something being computed differently.  Other times, I get warnings or errors.  I see most of the problems with LSTM’s, and in searching I saw a few bugs with LSTM’s and DataParallel.  Is it / should it be pretty safe to run things using DataParallel or is it unstable?  I am running 1.5.0a0+8f84ded, which I know there are newer versions, but I am running the NVIDIA NGC image and they haven’t released a new one.  I did update pytorch via pip just to try it out and still had some of the issues so I went back to 1.5.0a0+8f84ded.  Many times models don’t seem all that much faster with DataParallel, not sure if there are any tips to make things faster.  I do use pin_memory and calibrate my dataloader num_workers to find what is fastest.