Error in get embedding_size.
### System Info

- transformers version: 4.28.1
- deepspeed 0.8.3

In [examples](https://github.com/huggingface/transformers/tree/main/examples)/[pytorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch)/[language-modeling](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling)/[run_clm.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py)

when I set deepspeed config, I get an embedding_size as 0.




### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
embedding_size = model.get_input_embeddings().weight.shape[0]
if len(tokenizer) > embedding_size:
    model.resize_token_embeddings(len(tokenizer))
```

At this point, the value of embedding_size obtained is 0, which may cause an error in code. Maybe it's because using deepspeed config. 

```
{
  "fp16": {
    "enabled": "auto",
    "loss_scale": 0,
    "loss_scale_window": 1000,
    "initial_scale_power": 16,
    "hysteresis": 2,
    "min_loss_scale": 1
  },
  "optimizer": {
    "type": "AdamW",
    "params": {
      "lr": "auto",
      "betas": "auto",
      "eps": "auto",
      "weight_decay": "auto"
    }
  },
  "scheduler": {
    "type": "WarmupDecayLR",
    "params": {
      "warmup_min_lr": "auto",
      "warmup_max_lr": "auto",
      "warmup_num_steps": "auto",
      "total_num_steps": "auto"
    }
  },
  "zero_optimization": {
    "stage": 3,
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 1e9,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_max_live_parameters": 1e9,
    "stage3_max_reuse_distance": 1e9,
    "stage3_gather_16bit_weights_on_model_save": "auto"
  },
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "steps_per_print": 2000,
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "wall_clock_breakdown": false
}
```

### Expected behavior

Gets that the embedding size is not equal to zero
cc @stas00 <|||||>With zero-3 outside of fwd/bwd logic where this is done automatically you need to manually gather the sharded model's weights that you need. 

Please see:
https://huggingface.co/docs/transformers/main/main_classes/deepspeed#gathering-parameters

And you will find several examples in our code, e.g.:

https://github.com/huggingface/transformers/blob/88399476c3892435395618ed37993176dbb0de73/src/transformers/modeling_utils.py#L1455-L1456