[s2s] Trainer vs PTL timings
For the following two commands,

+ PTL finishes: 2.01 it/s, ~3H, 21.32 Rouge
+ Trainer: 1.0 it/s, roughly 5.5H, 21.36 Rouge

I wanted to report this so I don't lose track of it. Looked at the code, and don't see any obvious issue, besides that the slowdown is suspiciously close to 2x.



Any idea @patil-suraj ?

### PTL Command

```bash

export BS=32
export GAS=1

python finetune.py \
    --learning_rate=3e-5 \
    --fp16 \
    --gpus 1 \
    --do_train \
    --do_predict \
    --val_check_interval 0.25 \
    --n_val 500 \
    --num_train_epochs 2 \
    --freeze_encoder --freeze_embeds --data_dir cnn_dm \
    --max_target_length 142 --val_max_target_length=142 \
    --train_batch_size=$BS --eval_batch_size=$BS --gradient_accumulation_steps=$GAS \
    --model_name_or_path sshleifer/student_cnn_12_6 \
    --tokenizer_name facebook/bart-large \
    --warmup_steps 500 \
    --output_dir distilbart-cnn-12-6

```


### Trainer command
same as `builtin_trainer/train_distilbart_cnn.sh`:

```bash
export BS=32
export GAS=1
export m=sshleifer/student_cnn_12_6
export tok=facebook/bart-large
export MAX_TGT_LEN=142

python finetune_trainer.py \
    --model_name_or_path $m --tokenizer_name $tok \
    --data_dir cnn_dm \
    --output_dir distilbart-cnn-12-6-trainer --overwrite_output_dir \
    --learning_rate=3e-5 --sortish-sampler \
    --warmup_steps 500 \
    --fp16 \
    --n_val 500 \
    --gradient_accumulation_steps=$GAS \
    --per_device_train_batch_size=$BS --per_device_eval_batch_size=$BS \
    --freeze_encoder --freeze_embeds \
    --num_train_epochs=2 \
    --save_steps 3000 --eval_steps 3000 \
    --logging_first_step \
    --max_target_length 142 --val_max_target_length $MAX_TGT_LEN --test_max_target_length $MAX_TGT_LEN \
    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate --sortish_sampler
```
I might be misreading progress bars. Running another test, will reopen if I can replicate. <|||||>PTL (device 1) using less GPU ram it seems:

![image](https://user-images.githubusercontent.com/6045025/97605462-08285c00-19e5-11eb-9bc9-fd7900cde6b6.png)


Progress bars (note that PTL/Bottom is per epoch):

![image](https://user-images.githubusercontent.com/6045025/97605841-838a0d80-19e5-11eb-9e2a-eb71722f7308.png)
<|||||>I'm also experiencing slow down on TPU's, didn't run the new changes on GPU yet. I"ll investigate this<|||||>Thx!<|||||>I've confirmed that builtin ~2x slower on 1 GPU than PTL. Same commands as above on a different machine. All the screenshots above are valid.<|||||>These seem to run at the same speed if you pass `--fp16_opt_level=O1` to pytorch-lightning. Verifying now and will post results in 5 hrs.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
