You are correct in your understanding and as described in the docs 132, the input is expected as [seq_len, batch, input_size] in the default setup.
If you want the batch dimension to be in dim0, you can specify batch_first=True.
As far as I know, this might however lead to worse performance if you are using the GPU and cuDNN.