@apaszke @smth  Hello!
As we understood in this topic DataParallel does not work correct with RNN hidden states, when batch_first=True. It is because batch_first option affects only input and output of RNN, but not hidden states. Maybe it’s better to add the following feature: if batch_first=True then input and output hidden states of rnn will have following dimension (batch, num_layers * num_directions, hidden_size) instead of (num_layers * num_directions, batch, hidden_size), which is now fixed for both True and False options of batch_first. So it will be easier to work with hidden states without boilerplate permutations everytime. And it will be consistent with input and output dimensions.
Another way is to expand DataParallel functionality and use different dimensions for each neural network input. Now it uses the same split dimension for each type of input.