PyTorch 1.5 has completely fixed the issues with RNN training and DataParallel.
It seems it has done so quite seamlessly.
No more gerrymandering being required
I confirmed this today, in a project involving bidir GRUS on speech mfccs.