[resume optimization] skip loading pretrained weights on resume
This is similar to what was discussed in https://github.com/huggingface/transformers/issues/9205, which proposed not to random init weights on `from_pretrained`, but this time it's about resume - currently we load pretrained weights and immediately drop them on resume from checkpoint in Trainer.

To solve this we, for example, could change examples:

1. to figure out the checkpoint immediately after we init `TrainingArguments` and just before model is created. 
2. then change `from_pretrained()` API to do keep everything as is, except loading the weights from `state_dict`, if say `skip_weights_load=True` is passed:

So the code becomes:
```

    if training_args.do_train:
        if last_checkpoint is not None:
            checkpoint = last_checkpoint
        elif os.path.isdir(model_args.model_name_or_path):
            checkpoint = model_args.model_name_or_path
        else:
            checkpoint = None

    model = AutoModelForSeq2SeqLM.from_pretrained(
        model_args.model_name_or_path, 
        [...],
        skip_weights_load=checkpoint is not None,
    )

    if training_args.do_train:
        train_result = trainer.train(resume_from_checkpoint=checkpoint)
```

Any flaws in my thinking?

@patrickvonplaten, @sgugger 
This can be achieved by just doing `model = AutoModelForSeq2SeqLM.from_config(config)` when the checkpoint is not None. I don't believe it will be much faster however as your analysis in #9205 pointed out to the random initialization as being the bottleneck.<|||||>> This can be achieved by just doing model = AutoModelForSeq2SeqLM.from_config(config) when the checkpoint is not None. 

From here, right?
https://github.com/huggingface/transformers/blob/88ac60f7b5f6d4b62245dc21653ea3d5db7d4935/src/transformers/models/auto/auto_factory.py#L362

Great idea!

Then this important part would be missed:
```
            with deepspeed.zero.Init():
                model = cls(config, *model_args, **model_kwargs)
```
I guess I need to add it to `from_config` anyway, which would solve this part

and also this won't be done:
```
   model.eval()
```
but the latter is probably redundant anyway.

> I don't believe it will be much faster however as your analysis in #9205 pointed out to the random initialization as being the bottleneck.

For huge models every saving counts! once you start working with models like t5-11b it's excruciatingly slow to wait for things to start.

Should I try one example and re-shuffle the order of the code?<|||||>Yes, we should try on one example first! Though the first step is to fix the `from_config` method of `AutoModel` :-)