I think you could maybe reshape the vector by multiplying the seq_len*output_dim reshape(batch_size, seq_len*output_dim) before you pass it to the linear layer. I do not know if it is the most appropriate way to go for it, also looking at how they deal with this in the huggingface transformer library might help.
If you try this method please let know about the results