[trainer]  speed issues: --fp16 doesn't improve speed, DP runs really slow 
Splitting from https://github.com/huggingface/transformers/issues/9156#issuecomment-747636108 where while running benchmarks for fairscale's sharded ddp support I noticed that there was almost no difference between having or not having `--fp16` to the training runtime.

Not sure whether this impacts trainer in general or just seq2seq finetune_trainer.py, but there is almost no speed improvements with adding `--fp16`. 

This is with pytorch-nightly.

Not sure whether this has to do with the recent autocast cache blowup fix https://github.com/pytorch/pytorch/issues/48049 (our corresponding issue https://github.com/huggingface/transformers/issues/8403) - part of the fix was to remove some of the cash that perhaps was essential and thus leads to this issue.

I will test with apex and compare.

Also as can be seen from: https://github.com/huggingface/transformers/issues/9179#issuecomment-747783879
DP is running really slow!

@sgugger 

Could you run the other examples script to check? On my end they result in a roughly x2 speedup but I'm not on pytorch-nightly.<|||||>Any recommendations and specific command lines that you use? 

I have been using the finetune and other scripts in seq2seq as a go-to scripts for testing, so I'm not quite experienced with the others. 

Thank you!<|||||>So on a finetune_trainer  setup `--fp16` is slower than w/o it - on either amp or apex, so that elimination the caching concern. (w/ pytorch nightly)

Caveat: The following tests aren't ideal since I have 1 fast and 1 slow card, but they should be consistent since the overall speed is always at the slowest card (with the exception of single gpu tests), so it's like having 2 slow cards.

### DDP

```
# baseline w/ --fp16
export BS=4; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1 PYTHONPATH=../../src USE_TF=0   python -m torch.distributed.launch --nproc_per_node=2 --master_port=9910  ./finetune_trainer.py --model_name_or_path sshleifer/distill-mbart-en-ro-12-4 --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler --src_lang en_XX --task translation --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 500

2020-12-17 16:04:18 | INFO | __main__ |   train_runtime = 27.9693

# --fp16 --fp16_backend apex

2020-12-17 16:01:14 | INFO | __main__ |   train_runtime = 30.0469

# --fp16 --fp16_backend amp

2020-12-17 16:06:41 | INFO | __main__ |   train_runtime = 29.5368

```

### DP

DP setup is about the same correlation - but runs twice as slow! (that looks wrong too!)

```

# baseline (no --fp16)
export BS=4; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1 PYTHONPATH=../../src USE_TF=0   python  ./finetune_trainer.py --model_name_or_path sshleifer/distill-mbart-en-ro-12-4 --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler --src_lang en_XX --task translation --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 500

2020-12-17 16:16:09 | INFO | __main__ |   train_runtime = 56.7522

# --fp16 --fp16_backend apex

2020-12-17 16:14:26 | INFO | __main__ |   train_runtime = 59.4309

# --fp16 --fp16_backend amp

2020-12-17 16:12:18 | INFO | __main__ |   train_runtime = 58.4406
```



### Single GPU (gtx-1070 slowest)

no improvement either with fp16

```
# baseline (no --fp16)
export BS=4; rm -r output_dir; CUDA_VISIBLE_DEVICES=1 PYTHONPATH=../../src USE_TF=0   python  ./finetune_trainer.py --model_name_or_path sshleifer/distill-mbart-en-ro-12-4 --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler --src_lang en_XX --task translation --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 500

2020-12-17 16:26:10 | INFO | __main__ |   train_runtime = 24.6995

# --fp16 --fp16_backend apex

2020-12-17 16:27:26 | INFO | __main__ |   train_runtime = 28.6601

# --fp16 --fp16_backend amp

2020-12-17 16:28:37 | INFO | __main__ |   train_runtime = 27.9687

```


### Single GPU (rtx-3090 fastest)

no improvement either with fp16

```
# baseline (no --fp16)
export BS=4; rm -r output_dir; CUDA_VISIBLE_DEVICES=0 PYTHONPATH=../../src USE_TF=0   python  ./finetune_trainer.py --model_name_or_path sshleifer/distill-mbart-en-ro-12-4 --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler --src_lang en_XX --task translation --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 500

2020-12-17 16:22:33 | INFO | __main__ |   train_runtime = 11.2534

# --fp16 --fp16_backend apex

2020-12-17 16:19:41 | INFO | __main__ |   train_runtime = 14.4828

# --fp16 --fp16_backend amp

2020-12-17 16:21:15 | INFO | __main__ |   train_runtime = 11.7265
```

rtx-3040 is so much faster than gtx-1070.

Also wrt single gpu test: slow card 100%, fast one 75% - so the trainer is not fast enough feeding data in for the latter.
<|||||>> Any recommendations and specific command lines that you use?

Normally the first command indicated in the README of each example folder should work well :-)<|||||>> Normally the first command indicated in the README of each example folder should work well :-)

The one I tried doesn't report speed. `text-classification/run_glue.py`. I'm not sure what method you were using to detect speed improvements.<|||||>Thanks for adding the speed metrics into the core trainer, with https://github.com/huggingface/transformers/pull/9198 I run benchmarks for run_glue in text-classification, the results are terrible speed-wise just like with the finetune_trainer - terrible as in all those things that are supposed to speed things up, slow things down instead:

```
# baseline
rm -r output_dir; PYTHONPATH=../../src USE_TF=0  python -m torch.distributed.launch --nproc_per_node=2 run_glue.py --model_name_or_path bert-base-cased --task_name MRPC --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --output_dir output_dir

12/18/2020 13:29:01 - INFO - __main__ -     train_runtime = 97.4012

# --fp16

12/18/2020 13:46:09 - INFO - __main__ -     train_runtime = 109.1225

# --sharded_ddp

12/18/2020 13:53:18 - INFO - __main__ -     train_runtime = 103.1887

# --fp16 --sharded_ddp

12/18/2020 13:50:57 - INFO - __main__ -     train_runtime = 113.9132
```

this is all w/ pt-nightly, since I have to use it to run rtx-3090 card

<|||||>In that case, it looks linked to your setup/env somehow. Maybe support for the RTX-3090 is not fully fledged? On my setup and env FP16 speeds up by a factor of x2 roughly for this script. Will test the sharded DDP variants on Monday.<|||||>if you could do the above 4 runs (https://github.com/huggingface/transformers/issues/9179#issuecomment-748338332) for comparison that would be great! I need to know if something is off with my setup since I'm trying to eval deepspeed. 

cuda-11.2 is out so hoping to get a normal support for rtx-3090 from pytorch really soon now.<|||||>Hi 
the command @stas00  mentioend above does not work for me, I am using last version of huggingface, could you tell me which version you used to test? please see my bug here https://github.com/huggingface/transformers/issues/9215 <|||||>@rabeehk: https://github.com/huggingface/transformers/issues/9156#issuecomment-748501582<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.