Thanks for the response  - but I do want to explicitly define the hidden states (I plan to develop a custom LSTM based architecture) and initialize them every time, and thats when the dataparallel doesn’t split along the batch. Is there a way to force Pytorch to do this?