I agree with you. There is nothing theoretical preventing us from doing it.
I said it was a roadblock in the sense that it will be time consuming to implement and unlikely to be high priority enough to be done now by the core team.
That being said, we will be very happy to accept a PR that adds support for weights that have an extra dimension for the batch size.
If you do have time to work on this, you can open an issue that explains how you plan to add this and I will be more than happy to help have a good design for this and then review the implementation !
The tricky parts/questions that need answering are:
backend:

Fast conv for weights with an extra batch dimension. like mm/bmm, have conv/bconv
Check other commonly used Module (like batchnorm) to make sure the underlying implementation will support batched weights

frontend:

Do you need the ability to switch between the two modes?
Do we want this to be built in existing nn.Modules? (if so, how do we handle possibly changing batch-sizes)
Do we want element-wise versions of the existing modules as new Modules?