Out of Memory Error (OOM) only during evaluation phase of run_lm_finetuning.py and run_glue.py
## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): roberta-large

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* the official example scripts: run_lm_finetuning.py and run_glue.py

The tasks I am working on is:
* my own task or dataset: my own dataset which I have put into the correct format for `run_lm_finetuning` and the `SST-2` task for `run_glue.py`

## To Reproduce

When I use the official `run_lm_finetuning.py` and `run_glue.py` scripts with `roberta-large` on my own data, I run into out of memory (`OOM`) issues during `evaluation`. `Training completes successfully` with a `batch_size` of `1` and a `max_senquence_length` of `512`. However, when `evaluating` the 4th or 5th checkpoint, I get an `OOM` error. I tested using `distributed` training with `FP16` on `2 Tesla P100-SXM2 16 GB` and `2 Tesla P100-PCIE 12 GB` with the same results. I'm currently testing on `2 Tesla M40 24GB`; however, I don't always have access to these GPUs; so,  this is untenable long term.

## Expected behavior

Since it finishes training and evaluation of the first few checkpoints with the aforementioned setting, I would expect it to be able to evaluate every checkpoint without error. However, it seems that there is a bug in the evaluation code. I assume it is not properly clearing the memory after testing each checkpoint. 

## Environment

* OS: Ubuntu 18.04
* Python version: 3.6.8
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU Yes,  Tesla M40 24GB, Tesla P100-SXM2 16 GB, Tesla P100-PCIE 12 GB
* Distributed of parallel setup 2x of each separate GPU

Have you tried the following lines after the call to the `evaluate` function (in the loop over checkpoints)?

```python
del model
torch.cuda.empty_cache()
```

Let me know if this works.<|||||>@rlouf, I changed

```
# Evaluation
    results = {}
    if args.do_eval and args.local_rank in [-1, 0]:
        checkpoints = [args.output_dir]
        if args.eval_all_checkpoints:
            checkpoints = list(os.path.dirname(c) for c in sorted(glob.glob(args.output_dir + '/**/' + WEIGHTS_NAME, recursive=True)))
            logging.getLogger("transformers.modeling_utils").setLevel(logging.WARN)  # Reduce logging
        logger.info("Evaluate the following checkpoints: %s", checkpoints)
        for checkpoint in checkpoints:
            global_step = checkpoint.split('-')[-1] if len(checkpoints) > 1 else ""
            prefix = checkpoint.split('/')[-1] if checkpoint.find('checkpoint') != -1 else ""
            
            model = model_class.from_pretrained(checkpoint)
            model.to(args.device)
            result = evaluate(args, model, tokenizer, prefix=prefix)
            result = dict((k + '_{}'.format(global_step), v) for k, v in result.items())
            results.update(result)

    return results
```
to
```
# Evaluation
    results = {}
    if args.do_eval and args.local_rank in [-1, 0]:
        checkpoints = [args.output_dir]
        if args.eval_all_checkpoints:
            checkpoints = list(os.path.dirname(c) for c in sorted(glob.glob(args.output_dir + '/**/' + WEIGHTS_NAME, recursive=True)))
            logging.getLogger("transformers.modeling_utils").setLevel(logging.WARN)  # Reduce logging
        logger.info("Evaluate the following checkpoints: %s", checkpoints)
        for checkpoint in checkpoints:
            global_step = checkpoint.split('-')[-1] if len(checkpoints) > 1 else ""
            prefix = checkpoint.split('/')[-1] if checkpoint.find('checkpoint') != -1 else ""
            
            model = model_class.from_pretrained(checkpoint)
            model.to(args.device)
            result = evaluate(args, model, tokenizer, prefix=prefix)
            del model
            torch.cuda.empty_cache()
            result = dict((k + '_{}'.format(global_step), v) for k, v in result.items())
            results.update(result)

    return results

```
which didn't help.

So, I looked through https://github.com/huggingface/transformers/issues/1742 and added

```
del model
gc.collect()
torch.cuda.empty_cache()
```

This didn't work either; so, I change the scheduler to the following while leaving

```
del model
gc.collect()
torch.cuda.empty_cache()
```

```
def warmup_linear_schedule(optimizer, warmup_steps, t_total, last_epoch=-1):
    def lr_lambda(step):
        if step < warmup_steps:
            return float(step) / float(max(1, warmup_steps))
        return max(0.0, float(t_total - step) / float(max(1.0, t_total - warmup_steps)))

    return LambdaLR(optimizer, lr_lambda, last_epoch=-1)

```
This hasn't worked either.

Any ideas? It doesn't seem like I can `del model`<|||||>Hi! Changing the scheduler should not do anything since you are in evaluation mode. This is likely not due to the model either, since `del` + `gc.collect()` should remove it from memory. Could you look at by how much the memory increases after each step?<|||||>So, when using `roberta-large` the memory increases by ~2.25 GB after each checkpoint starting at ~7GB.<|||||>Can you have a look at which tensors are still being tracked, using [this code](https://github.com/huggingface/transformers/issues/1742#issuecomment-550327172) after your del statement?<|||||>My apologies for the delay in my answer. I had hoped that https://github.com/huggingface/transformers/commit/9629e2c676efab31d1f27a3f27d811e91fd05a92 would solve my issue; however, evaluation doesn't appear to be distributed. So, it is still using one card and is still ~2.25 GB after each checkpoint. Should I open up a separate issue for the distributed evaluation issue? After adding 
```
def print_gpu_obj():
    import gc
    GPU_count = 0
    Pinned_count = 0
    for tracked_object in gc.get_objects():
        if torch.is_tensor(tracked_object):
            if tracked_object.is_cuda:
                GPU_count+=1
            if tracked_object.is_pinned():
                Pinned_count+=1
            

    print("There are {} cuda objects".format(GPU_count))
    print("There are {} pinned objects".format(Pinned_count))
```
to the code, I get

before `del model`
checkpoint 0
```
There are 1087 cuda objects
There are 0 pinned objects
```
checkpoint 1
```
There are 1673 cuda objects
There are 0 pinned objects
```
checkpoint 2
```
There are 2259 cuda objects
There are 0 pinned objects
```
checkpoint 3
```
There are 2845 cuda objects
There are 0 pinned objects
```

after `del model` and before `gc.collect`
checkpoint 0
```
There are 984 cuda objects
There are 0 pinned objects
```
checkpoint 1
```
There are 1570 cuda objects
There are 0 pinned objects
```
checkpoint 2
```
There are 2156 cuda objects
There are 0 pinned objects
```
checkpoint 3
```
There are 2742 cuda objects
There are 0 pinned objects
```

after `gc.collect` and before `torch.cuda.empty_cache()`  and after `torch.cuda.empty_cache()` are both the same as after `del model` and before `gc.collect`.

It appears that 586 objects are added at each checkpoint, which cannot be deleted and unallocated. Any suggestions?<|||||>This looks like some data is kept in memory after evaluation. Any way you could try to track this down?<|||||>My apologies for the delay in getting back to this, as well as my ignorance on this subject, but how would I track this down. As in what commands or tools should I use? Again sorry for the late reply, I got stuck on another project for a bit.<|||||>I have a similar issue. During training with run_lm_finetuning.py, the memory usage of a single GPU  is 30G/32G. And after the training stage, I mean at the beginning of eval, the memory doesn't drop down and the evaluation stage is always getting OOM. If I add torch.cuda.empty_cache() before evaluation, the memory drops down to 22G, which means there are still 22GB tensors exist but I don't know where they are and how to free them. If I do the evaluation with run_lm_finetuning.py without training, everything is ok and the memory usage is about 8G.<|||||>> I have a similar issue. During training with run_lm_finetuning.py, the memory usage of a single GPU is 30G/32G. And after the training stage, I mean at the beginning of eval, the memory doesn't drop down and the evaluation stage is always getting OOM. If I add torch.cuda.empty_cache() before evaluation, the memory drops down to 22G, which means there are still 22GB tensors exist but I don't know where they are and how to free them. If I do the evaluation with run_lm_finetuning.py without training, everything is ok and the memory usage is about 8G.

PS: If I set --evaluate_during_training, there is no OOM issue.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
