Regression in training speed since 4.4.0
## Environment info

- `transformers` version: 4.4.0/4.6.1
- Platform: Linux-5.8.0-53-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: Yes, RTX 3090
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger @patrickvonplaten

## Information

I've noticed a training speed regression between 4.4.0 and 4.6.0:

4.4.0 BS8:
{'train_runtime': 21.363, 'train_samples_per_second': 5.851, 'epoch': 1.0}
{'train_runtime': 21.6148, 'train_samples_per_second': 5.783, 'epoch': 1.0}
{'train_runtime': 21.7867, 'train_samples_per_second': 5.737, 'epoch': 1.0}  
 
4.6.1 BS8:
{'train_runtime': 23.7011, 'train_samples_per_second': 5.274, 'epoch': 1.0}    
{'train_runtime': 24.2845, 'train_samples_per_second': 5.147, 'epoch': 1.0}
{'train_runtime': 23.5801, 'train_samples_per_second': 5.301, 'epoch': 1.0}    
    
4.4.0 BS4:
{'train_runtime': 25.4107, 'train_samples_per_second': 9.838, 'epoch': 1.0}  
  
4.6.1 BS4:
{'train_runtime': 31.2902, 'train_samples_per_second': 7.99, 'epoch': 1.0}

I'm running the Pytorch 1.8.1. release, on my RTX3090/Ryzen 3700X workstation.

The performance loss seems to increase with smaller batch sizes, leading me to think it's something in Trainer.

Although I found the regression with Sequence Classification, I've found the the slowdown transfers to other tasks as well.

## To reproduce

```
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer

BATCH_SIZE = 4

raw_datasets = load_dataset("imdb")

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
full_train_dataset = tokenized_datasets["train"]
full_eval_dataset = tokenized_datasets["test"]

model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)

training_args = TrainingArguments("test_trainer", num_train_epochs=1, per_device_train_batch_size=BATCH_SIZE)

trainer = Trainer(
    model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset
)

trainer.train()
```
Thanks for flagging! I used your reproducer (very nice by the way!) and a git bisect and it comes from #10937, which reworks the way memory metrics are computed. You just have to set `skip_memory_metrics=True` in your `TrainingArguments` to skip the memory metric computation and you will get your performance back.<|||||>Thanks, that did the trick! Turns out that setting skip_memory_metrics also improves performance ~5% on 4.4.0.

Could I suggest that this parameter is enabled by default? It seems to me that this is debugging functionality, and shouldn't be enabled normally.<|||||>Yes, this is done by the PR mentioned above!<|||||>Quick comment on the already closed issue. Had to debug this issue independently, due to an older fork. 

It seems that the the issue is not just slowdown when enabling memory metrics, but also, there is performance variability from run to run. 

One of the signatures of the issue is that there is no performance loss or variability in backward() call (run in the Autograd C engine). The optimizer.step() had the greatest impact, followed by forward propagation. Based on those observations, the issue is suspected to be due to fast I/O ( gpu kernel calls during optimizer.step() and forward) affecting multithreading in the Python GIL. 

Skipping memory metrics fixes issue sufficiently. There is still a logging thread, and a progress-bar (tqdm) thread. Adding this note here as a warning that multithreading during forward or optimizer.step() might cause performance loss/variability. <|||||>Ouch, hope that it didn't cost you too much time! Thanks for the further info on the problem