@Varg_Nord I found the problem. If batch_first=True  is used, then DataParallel with default parameter dim=0 will split input_var and h0 in first dimension. It is correct for the input_var, but not for h0, because rnn hidden states always have dimension is equal to num_layers * num_directions x batch_size x hidden_size. For easy solution you can use batch_first=False, so the second dimension will correspond to batch dimension in input_var and use DataParallel with dim=1, so it will split input_var and h0 using correct dimension for both variables.
If you want to use batch_first=True, then you can swap axes for hidden states before .forward and swap back inside .forward. But it is computationally inefficient.