Thanks for the clarification! I am indeed using DataParallel()
Please see the edit I made. The split batch dimensions are getting coalesced for the output from my LSTM encoder, but for the hidden features the dimensions are not being coalesced in the same way. The hidden features are not being returned as 1x16x128 (num_layers x batch_size xhidden_size) but instead as 4x4x128 on multi gpu training.