Checkpoint sharding
# What does this PR do?

This PR introduces the ability to create and load sharded checkpoints. It introduces a new argument in `save_pretrained` that controls the maximum size of a checkpoint before being auto-sharded into smaller parts (which defaults to 10GB after internal discussion, which should be good with the Hub and environment with low RAM like Colab).

When the model total size is less than this maximum size, it's saved exactly like before. When the model size is bigger, while traversing the state dict, each time a new weight tips the size above that threshold, a new shard is created. Therefore each shard is usually of size less than the max size, but if an individual weight has a size bigger than this threshold, it will spawn a shard containing only itself that will be of a bigger size.

On the `from_pretrained` side, a bit of refactoring was necessary to make the API deal with several state dict files. The main part is isolating the code that loads a state dict into a model in a separate function, so I can call it for each shard. I'm leaving comments on the PR to facilitate the review and I will follow up with another PR that refactors `from_pretrained` even more for cleaning but with no change of actual code.

cc @julien-c @thomwolf @stas00 @Narsil who interacted in the RFC.

Linked issue: #13548 
_The documentation is not available anymore as the PR was closed or merged._<|||||>@sgugger is it possibile to apply sharding to a current pretrained model that is bigger than 10gb (let's say t0pp like), using 

load_pretrained-> save_pretrained

To save a sharded version, and the again

load_pretrained

That should handle therefore load the sharded checkpoint?

Thanks<|||||>Yes, that's exactly what you should do!
We'll also create a new branch of the t0pp checkpoint with a sharded checkpoint (we can't do it on the main branch or it would break compatibility with older versions of Transformers).<|||||>Hahaha amazing! Where is it? <|||||>@sgugger we could possibly upload the sharded checkpoint **in addition to** the current checkpoint in the same repo branch no?

I thought that's what we wanted to do to preserve backward compat while still upgrading those big models
<|||||>If you put them in the same branch, `from_pretrained` will only download the full checkpoint as it's the first in order of priority.<|||||>I see. should we consider changing this behavior? we've seen that git-branches are not super practical for those large models (cc @osanseviero )<|||||>(yep, i know this is going to require to call something like /api/models/xxx at the start of `from_pretrained`... :) )<|||||>or maybe we can say that we'll just push sharded models from now on (for example the bigscience models will be sharded-only) I think that's actually fine<|||||>or we can default to the shared model for the new version of `transformers`?<|||||>yep that was what i was suggesting but implementation wise is a bit more complex (and affects all model repos not just the sharded ones)