CUDA out of memory error on Trainer hyperparameter_search
Hi, 

I am using Colab to GridSearch on a dataset. The dataset has 7000 samples. I use 1/5 shard and batch_size is 1. No matter what I do, I get this error.

Env Related Info
-----------
tranformers (4.2.1)
datasets (1.2.1)

Model Used
----------------
[SpanBERT Large](https://huggingface.co/SpanBERT/spanbert-large-cased)

Snippets
------------
My dataset has 7934  train examples and 690  eval examples, with maximum number of tokens around 300 per example.

Tokenization Details:
```
  max_length: 384
  doc_stride: 128
```
```
def model_init():
    return AutoModelForQuestionAnswering.from_pretrained(model_checkpoint_name, num_labels=2)

args = TrainingArguments(
    output_dir = os.path.join(checkpoint_dir,'/grid_search/'),
    logging_dir= os.path.join(runs_dir,'/grid_search/'),
    evaluation_strategy='epoch',
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    gradient_accumulation_steps=1,
    learning_rate=2e-5,
    weight_decay=0.01,
    num_train_epochs=3.0,
    lr_scheduler_type="linear",
    warmup_steps=0,
    logging_steps=200,
    save_steps=200,
    seed=42
)

grid_search_trainer = Trainer(
    model_init=model_init,
    args=args,
    train_dataset=tokenized_train.shard(index=1,num_shards=5),
    eval_dataset=tokenized_val.shard(index=1,num_shards=5),
    data_collator=data_collator,
    tokenizer=tokenizer
)

best_run = grid_search_trainer.hyperparameter_search(n_trials=10, direction="minimize")
```

Sometimes, this happens after the trainer is done with one epoch. Is the trainer initializing the model without deleting/clearing the previous one? Would that affect the GPU? How can I prevent this issue?

![image](https://user-images.githubusercontent.com/29076344/105278992-a551f580-5bcc-11eb-9056-91560a6a99d4.png)

@patil-suraj 
There's not enough information here to help you. Please provide env info, the model/tasks you are using, and a short text code snippet to reproduce the error.  <|||||>@patil-suraj I have edited the comment. Please help me out. Thanks.<|||||>@patil-suraj Same issue here
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.