Actually it is seq_len x batch_size. I printed shapes for all layers:
inputs shape:  torch.Size([377, 2])
embeds shape:  torch.Size([377, 2, 100])
hidden[0] shape:  torch.Size([2, 64])
output shape:  torch.Size([2, 126])