Inference time got very high, very low CUDA activity
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.0.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: no

## Information

I am using Trainer() for some BERT/DistilBERT experiments.
After I upgraded to the latest git master version, the inference time got very high. 
Noted that I have very low CUDA activity (CPU is also very low).
Seems to be something messed up around training.


The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. pip install transformers==4.4.2
2. Run script below, and inspect inference time, CUDA activity.
3. pip install git+https://github.com/huggingface/transformers 
4. Run script below, and inspect inference time, CUDA activity (note that there is no/or very low CUDA activity, and inference time got very high).

`from transformers import Trainer, TrainingArguments
from transformers import DistilBertForSequenceClassification, DistilBertTokenizerFast
from datasets import load_dataset
from sklearn.metrics import accuracy_score, precision_recall_fscore_support

select_model = 'distilbert-base-uncased'
model = DistilBertForSequenceClassification.from_pretrained(select_model, num_labels=2, force_download=False)
tokenizer = DistilBertTokenizerFast.from_pretrained(select_model, force_download=False)
#model.config

train_dataset, val_dataset = load_dataset('imdb', split=['train[:20%]', 'test[:20%]'])
train_dataset = train_dataset.rename_column('text','sentence')
val_dataset = val_dataset.rename_column('text','sentence')

sentence_lenght = 5

def tokenize(batch):
    return tokenizer(batch['sentence'], padding=True, truncation=True, max_length=sentence_lenght)  # batch['text']

train_dataset = train_dataset.map(tokenize, batched=True, batch_size=len(train_dataset))
val_dataset = val_dataset.map(tokenize, batched=True, batch_size=len(val_dataset))
train_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'label'])
val_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'label'])

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary')
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

training_args = TrainingArguments(
    run_name='experiments_distilBert_01',
    output_dir='./results',
    num_train_epochs=4,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    warmup_steps=500,
    learning_rate=2e-5,
    weight_decay=0.01,
    evaluation_strategy="epoch",
    logging_dir='./logs',
    logging_steps=250,
    save_strategy='no',
    report_to=['tensorboard'],
    # deepspeed='./ds_config.json'
    fp16=False,
    fp16_backend='auto',
    disable_tqdm=False,
    load_best_model_at_end=True
)

trainer = Trainer(
    model=model,
    args=training_args,
    tokenizer=tokenizer,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    compute_metrics=compute_metrics
)

trainer.train()
trainer.evaluate()`

## Expected behavior

Scripts execute at least with the same inference time.

I'm having a similar issue as well but occurring at both test and training time - <3% GPU utilization. When executing the `run_qa.py` script using the command line arguments in the first example of the question answering example, it takes much longer than when I was running `transformers` v4.3.3.

However, the script seems to run fine on our cluster (also running v4.5.0) using 2080Ti's.

- transformers version: 4.5.0
- Platform: Windows 10 Version 2004 (OS Build 19041.264)
- Python version: 3.9,2
- PyTorch version (GPU?): 1.8.1 (RTX 3090)
- Tensorflow version (GPU?): None
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: no
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Installing the latest 4.6.0.dev0 dev release from git is still very slow on GoogleColab, compared to the latest official when running text classification on BERT, distillBert or Xlm-r.<|||||>I have the same issue.<|||||>Hello! You mention Google Colab, do you have a notebook to share so that we can take a look? Thank you.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.