Gradient is None in after deepspeed backward 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->


- `transformers` version: 4.5.1
- Platform: Linux-4.15.0-112-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: GeForce RTX 3090
- Using distributed or parallel set-up in script?: Deepspeed

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. !-->

- @stas00 
- @sgugger 

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] my own task or dataset: (give details below)

I want to check if the gradient in different GPU process are same after backward, so I output the gradient in trainer.py:
![image](https://user-images.githubusercontent.com/15072042/119375775-ce56d400-bced-11eb-8fa6-e59438390faa.png)

The output shows that all gradients are None. I reproduced this in another script that I believe working as expected for a long time. So my quesion is:
1. is this by design? why it's None.
2. how to output the real gradient of the model's parameters after gradient are calcuated? 

deepspeed config:

```

    "gradient_accumulation_steps": 1,
    "train_batch_size": 16,
    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "hysteresis": 2,
        "min_loss_scale": 1,
        "initial_scale_power": 16
    },

    "zero_optimization": {
        "stage": 2,
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": 2e8,
        "contiguous_gradients": true,
        "cpu_offload": false,
        "find_unused_parameters": true
    },

    "zero_allow_untested_optimizer": true,

    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": 2e-5,
            "betas": [0.9, 0.999],
            "eps": 1e-6,
            "weight_decay": 0.01 
        }
    },

    "scheduler": {
        "type": "OneCycle",
        "params": {
            "cycle_first_step_size": 5000,
            "cycle_first_stair_count": 500,
            "cycle_second_step_size": 5000,
            "cycle_second_stair_count": 500,
            "decay_step_size": 1000,
            "cycle_min_lr": 4e-5,
            "cycle_max_lr": 1e-4,
            "decay_lr_rate": 0.001,
            "cycle_momentum": true,
            "cycle_min_mom": 0.85,
            "cycle_max_mom": 0.99,
            "decay_mom_rate": 0.0
        }
    },

    "steps_per_print": 500,
    "wall_clock_breakdown": false
}
```


**edit** looks like Deepspeed needs to add an API to do that: https://github.com/microsoft/DeepSpeed/issues/1098

My original suggestion is very likely not to work. I have just never tried it in this context:

---------

Since params are sharded, you need to gather them before you can read their values. https://deepspeed.readthedocs.io/en/latest/zero3.html#gathering-parameters

Here is an untested code:
```
import deepspeed
for name, param in model.named_parameters():
    with deepspeed.zero.GatheredParameters(param, modifier_rank=None):
        if param.requires_grad: ....
```
<|||||>meanwhile please edit the OP to include the ds config file you used, so that we know what setup you're running it under.<|||||>> **edit** looks like Deepspeed needs to add an API to do that: [microsoft/DeepSpeed#1098](https://github.com/microsoft/DeepSpeed/issues/1098)
> 
> My original suggestion is very likely not to work. I have just never tried it in this context:
> 
> Since params are sharded, you need to gather them before you can read their values. https://deepspeed.readthedocs.io/en/latest/zero3.html#gathering-parameters
> 
> Here is an untested code:
> 
> ```
> import deepspeed
> for name, param in model.named_parameters():
>     with deepspeed.zero.GatheredParameters(param, modifier_rank=None):
>         if param.requires_grad: ....
> ```

thanks for asking this feature in deepspeed and confrimed None gradient is expected for now<|||||>> meanwhile please edit the OP to include the ds config file you used, so that we know what setup you're running it under.

