[DeepSpeed ZeRO3] Fix performance degradation in sharded models
When sharded models were added the deepspeed/zero3 branch of model loading of pretrained weights 

https://github.com/huggingface/transformers/blob/7d5fde991d598370d961be8cb7add6541e2b59ce/src/transformers/modeling_utils.py#L427-L429

got ~N-shards-x-slower, since it wastefully gathered weights that weren't in the `state_dict`:

**So for example with BLOOM-176 which has 72 shards, the loading was ~70x slower under deepspeed zero3 and nvme offload!**

This fix takes care of the situation with sharded models by finding an intersection of `state_dict` keys and the keys of the parameters of the current submodule that is being loaded to and thus only gathering the weights that get updated. And if there is no intersection skipping the rest of the branch altogether.

The 1-shard use case still works as the intersection should be 100%.


_The documentation is not available anymore as the PR was closed or merged._<|||||>@sgugger, I was thinking about this discovery and my suggestion is that we do this not just for deepspeed.

The thing is - with sharded models like bloom-176 (72 shards) - 90% of the time this code:

https://github.com/tjruwase/transformers/blob/81dbd0ba6c5422dec33dd31cf076e44d96d2d968/src/transformers/modeling_utils.py#L436

is doing nothing since its `state_dict`'s "payload" doesn't match the params of the submodule it's called for, as most of the time they are in another shard.

Not sure of the cost or the actual promised saving, but it will save many unnecessary `module._load_from_state_dict(*args)` calls.

Especially since the code is already there anyway.

Thoughts?<|||||>Yes, we could probably ignore the load when there is no parameter to load indeed. Will make a PR this morning.<|||||>Took a stab at it in #18937. Thinking more of it, I'm not sure if we'll get a sensible gain as I expect the calls to `module._load_from_state_dict` to be mostly noops but we can certainly train and measure the difference!<|||||>I followed up here: https://github.com/huggingface/transformers/pull/18937#pullrequestreview-1100945027
