Good obserations. Two issues have been created in github to track the suggestions here:
https://github.com/pytorch/pytorch/issues/30348 24 -> Speed up calc q params in observers.
https://github.com/pytorch/pytorch/issues/30349 19 -> Speed up per channel fake-quant
For fake-quant it will be better to speed up the C++ implementation to parallelize it.