So after asking around, basically, hidden_state is just not influenced by the batch_first argument. So it will have this shape whatever you pass.
This inconsistency is tracked in the following issue 24.