Your output should be returned with the shape [batch, seq_len, num_directions * hidden_size] and it seems you are directly feeding this input to the linear layer.
Note that linear layers accept inputs as [batch_size, *, nb_features], where the asterisk can be any dimensions, such that the linear layer will be applied on each of these dimensions separately.
I’m not sure what your exact use case is, but you could use:
output.view(batch, seq_len, num_directions, hidden_size)

to separate the direction dimension, if you need to slice the output tensor.