Very slow process when `torch_dtype` is passed.

When I use `from_pretrained`, the model loads up from my azure cache at almost 1gbps, but when I specify the `torch_dtype` the process is slowed to a crawling 1 tenth of original speed. When looking at resources it appears that its a single core type of process that is doing the disservice of being bottleneck.

Can we have that parallelized?
There is nothing we can do without a clear reproducer.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.