Slower training speed under DeepSpeed
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.8.2
- Platform: nvcr.io/nvidia/pytorch:21.02-py3 container on CentOS7
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0 with GPU
- Tensorflow version (GPU?): not used
- Using GPU in script?: YES, Tesla P40
- Using distributed or parallel set-up in script?: YES

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: 
   examples/pytorch/translation/run_translation.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: 
   wwm16
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create container using `sudo docker run -d -it --runtime=nvidia --net=host --ipc=host -v /home/user/:/workspace nvcr.io/nvidia/pytorch:21.02-py3  bash` on a Linux server with CentOS7 and Tesla P40 GPUs.
2. Install python dependencies mentioned above.
3. Run run_translation.py with different parameters listed below:
    1. DDP with fp16 open
     CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node 4 examples/pytorch/translation/run_translation.py  --model_name_or_path t5-small --per_device_train_batch_size 1   --output_dir output_dir --overwrite_output_dir --fp16 --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro

    2. DDP without fp16
    CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node 4 examples/pytorch/translation/run_translation.py  --model_name_or_path t5-small --per_device_train_batch_size 1   --output_dir output_dir --overwrite_output_dir --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro ```

     3. DeepSpeed ZeRO2 with fp16 open
     deepspeed examples/pytorch/translation/run_translation.py --deepspeed tests/deepspeed/ds_config_zero2.json --model_name_or_path t5-small --per_device_train_batch_size 1   --output_dir output_dir --overwrite_output_dir --fp16 --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro

    4. DeepSpeed ZeRO2 without fp16
    deepspeed examples/pytorch/translation/run_translation.py --deepspeed tests/deepspeed/ds_config_zero2.json --model_name_or_path t5-small --per_device_train_batch_size 1   --output_dir output_dir --overwrite_output_dir --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro

    5. DeepSpeed ZeRO3 with fp16 open
    deepspeed examples/pytorch/translation/run_translation.py --deepspeed tests/deepspeed/ds_config_zero3.json --model_name_or_path t5-small --per_device_train_batch_size 1   --output_dir output_dir --overwrite_output_dir --fp16 --do_train --max_train_samples 500 --num_train_epochs 1 --dataset_name wmt16 --dataset_config "ro-en" --source_lang en --target_lang ro

4. Training metrics are listed according to the above experiments order:
     1. ***** train metrics *****
  epoch                    =        1.0
  train_loss               =     1.5905
  train_runtime            = 0:00:20.29
  train_samples            =        500
  train_samples_per_second =     24.632
  train_steps_per_second   =      6.158
    2. ***** train metrics *****
  epoch                    =        1.0
  train_loss               =      1.482
  train_runtime            = 0:00:17.57
  train_samples            =        500
  train_samples_per_second =     28.448
  train_steps_per_second   =      7.112
    3. ***** train metrics *****
  epoch                    =        1.0
  train_loss               =     1.6752
  train_runtime            = 0:00:32.45
  train_samples            =        500
  train_samples_per_second =     15.406
  train_steps_per_second   =      3.851
    4. ***** train metrics *****
  epoch                    =        1.0
  train_loss               =      1.523
  train_runtime            = 0:00:20.15
  train_samples            =        500
  train_samples_per_second =     24.813
  train_steps_per_second   =      6.203
    5. ***** train metrics *****
  epoch                    =        1.0
  train_loss               =      1.523
  train_runtime            = 0:00:20.15
  train_samples            =        500
  train_samples_per_second =     24.813
  train_steps_per_second   =      6.203
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
According to DeepSpeed's official document, its training speedup could up to 10 times faster. But in my experiments I could not get that much speedup. Since translation script is the tutorial task mentioned in the Transformers's "DeepSpeed Integration"  document, so my expectation is a faster training speed. Is there any environment's limitations in my experiment? Or the speedup is not guaranteed? Thank you guys in advance for help me out. 
<!-- A clear and concise description of what you would expect to happen. -->

@stas00 
Deepspeed is a project that has many different at times totally unrelated features.

Therefore when you read in a blog that it made something 10x faster, you need to pay close attention to what was the task, and what was the model size, and how many hundreds of gpus, and optimizer, etc., etc.

The main goal of deepspeed is to enable training huge models which is not possible using bare pytorch. In particular when you can't fit your model onto a single GPU. Which means a lot more overhead. Therefore if you're going to compare a straightforward bare-bones pytorch to any other complex solution that enables scalabilty the former will almost always be faster or on par.

Then as I started this comment Deepspeed has other tools, like faster optimizers,  like 1-bit adam as posted here: https://www.deepspeed.ai/news/2020/09/08/onebit-adam-blog-post.html, which you haven't been using in your test. 

I hope this gave you a little bit of clarity of what to expect when.

We have only the main functionality integrated and lots of features are still pending as you can see here https://github.com/huggingface/transformers/issues/9606 - some of them probably require no integration but need to be tested, we just haven't had the time to work on those yet. And I think that list is far from being complete, since the Deepspeed team adds new features all the time.

If you're interesting in particular in a specific feature please first try and see if it already works with transformers/HF Trainer, if not, let's discuss the feasibility of its integration.<|||||>> Deepspeed is a project that has many different at times totally unrelated features.
> 
> Therefore when you read in a blog that it made something 10x faster, you need to pay close attention to what was the task, and what was the model size, and how many hudreds of gpus, and optimizer, etc., etc.
> 
> The main goal of deepspeed is to enable training huge models which is not possible using bare pytorch. In particular when you can't fit your model onto a single GPU. Which means a lot more overhead. Therefore if you're going to compare a straightforward barebones pytorch to any other complex solution that enables scalabilty the former will almost always be faster or on par.
> 
> Then as I started this comment Deepspeed has other tools, like faster optimizers, like 1-bit adam as posted here: https://www.deepspeed.ai/news/2020/09/08/onebit-adam-blog-post.html, which you haven't been using in your test.
> 
> I hope this gave you a little bit of clarity of what to expect when.
> 
> We have only the main functionality integrated and lots of features are still pending as you can see here #9606 - some of them probably require no integration but need to be tested, we just haven't had the time to work on those yet. And I think that list is far from being complete, since the Deepspeed team adds new features all the time.
> 
> If you're interesting in particular in a specific feature please first try and see if it already works with transformers/HF Trainer, if not, let's discuss the feasibility of its integration.

Thank you for your reply Stas, I've been more clarified after read your detailed comment. I'll checkout more features and give you feedback if any progress.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.