Resource exhausted when training in loop
## Environment info     
- `transformers` version: 3.4.0
- Platform: Linux-4.15.0-109-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik , @thomwolf or @jplu maybe?

## Information

Model I am using Bert:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce
To reproduce this error, one can use the script below. I have taken it from a blog post I made and altered it a bit - so it is a bit long. The main thing however, seems to be that if I load and fit a model several times it will at some point make my GPU run out of memory. Here I use kerastuner for hyperparam optimization and after 7 trials my GPU runs out of memory and thoughs the error below:
`  (0) Resource exhausted:  OOM when allocating tensor with shape[3200,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc         [[node BERT_MultiLabel_MultiClass/tf_bert_model/bert/encoder/layer_._11/intermediate/dense/Tensordot/MatMul (defined at myLib/python3.7/site-packages/transformers/modeling_tf_bert.py:327) ]]
`

### Reproducible script
```
#######################################
### -------- Load libraries ------- ###

# Load Huggingface transformers
from transformers import TFBertModel, BertConfig, BertTokenizerFast

# Then what you need from tensorflow.keras
from tensorflow.keras.layers import Input, Dropout, Dense
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.initializers import TruncatedNormal
from tensorflow.keras.losses import CategoricalCrossentropy
from tensorflow.keras.metrics import CategoricalAccuracy
from tensorflow.keras.utils import to_categorical

# And pandas for data import + sklearn because you allways need sklearn
import pandas as pd
from sklearn.model_selection import train_test_split


#######################################
### --------- Import data --------- ###

# Import data from csv
data = pd.read_csv('dev/Fun with BERT/complaints.csv')

# Select required columns
data = data[['Consumer complaint narrative', 'Product', 'Issue']]

# Remove a row if any of the three remaining columns are missing
data = data.dropna()
data = data.sample(n = 5000)

# Remove rows, where the label is present only ones (can't be split)
data = data.groupby('Issue').filter(lambda x : len(x) > 1)
data = data.groupby('Product').filter(lambda x : len(x) > 1)

# Set your model output as categorical and save in new label col
data['Issue_label'] = pd.Categorical(data['Issue'])
data['Product_label'] = pd.Categorical(data['Product'])

# Transform your output to numeric
data['Issue'] = data['Issue_label'].cat.codes
data['Product'] = data['Product_label'].cat.codes

# Split into train and test - stratify over Issue
data, data_test = train_test_split(data, test_size = 0.2, stratify = data[['Issue']])


#######################################
### --------- Setup BERT ---------- ###

# Name of the BERT model to use
model_name = 'bert-base-uncased'

# Max length of tokens
max_length = 100

# Load transformers config and set output_hidden_states to False
config = BertConfig.from_pretrained(model_name)
config.output_hidden_states = False

# Load BERT tokenizer
tokenizer = BertTokenizerFast.from_pretrained(pretrained_model_name_or_path = model_name, config = config)



#######################################
### ------- Build the model ------- ###

def model_build(hp):
    # TF Keras documentation: https://www.tensorflow.org/api_docs/python/tf/keras/Model

    # Load the Transformers BERT model
    transformer_model = TFBertModel.from_pretrained(model_name, config = config)

    # Build your model input
    input_ids = Input(shape=(max_length,), name='input_ids', dtype='int32')
    # attention_mask = Input(shape=(max_length,), name='attention_mask', dtype='int32') 
    # inputs = {'input_ids': input_ids, 'attention_mask': attention_mask}
    inputs = {'input_ids': input_ids}

    # Load the Transformers BERT model as a layer in a Keras model
    bert_model = transformer_model(inputs)[1]
    dropout = Dropout(hp.Float('hp_dropout', min_value=0, max_value=0.99, default = 0.2), name='pooled_output')
    pooled_output = dropout(bert_model, training=False)

    # Then build your model output
    issue = Dense(units=len(data.Issue_label.value_counts()), kernel_initializer=TruncatedNormal(stddev=hp.Float('hp_stddev_issue', min_value=0, max_value=1, default = 0.02)), name='issue')(pooled_output)
    product = Dense(units=len(data.Product_label.value_counts()), kernel_initializer=TruncatedNormal(stddev=hp.Float('hp_stddev_product', min_value=0, max_value=1, default = 0.02)), name='product')(pooled_output)
    outputs = {'issue': issue, 'product': product}

    # And combine it all in a model object
    model = Model(inputs=inputs, outputs=outputs, name='BERT_MultiLabel_MultiClass')

    # Set an optimizer
    optimizer = Adam(
        learning_rate=5e-05,
        epsilon=1e-08,
        decay=0.01,
        clipnorm=1.0)

    # Set loss and metrics
    loss = {'issue': CategoricalCrossentropy(from_logits = True), 'product': CategoricalCrossentropy(from_logits = True)}
    metric = {'issue': CategoricalAccuracy('accuracy'), 'product': CategoricalAccuracy('accuracy')}

    # Compile the model
    model.compile(
        optimizer = optimizer,
        loss = loss, 
        metrics = metric)
    
    return model


from kerastuner import HyperModel
class ClsModel(HyperModel):

    def build(self, fitting_param = None):
        
        model = model_build()

        return model



#######################################
### ------- Train the model ------- ###


# Ready output data for the model
y_issue = to_categorical(data['Issue'])
y_product = to_categorical(data['Product'])

# Tokenize the input (takes some time)
x = tokenizer(
    text=data['Consumer complaint narrative'].to_list(),
    add_special_tokens=True,
    max_length=max_length,
    truncation=True,
    padding=True, 
    return_tensors='tf',
    return_token_type_ids = False,
    return_attention_mask = True,
    verbose = True)

from kerastuner.tuners import BayesianOptimization
tuner = BayesianOptimization(
    model_build,
    objective='loss',
    max_trials=50,
    executions_per_trial=5,
    project_name='modeltest')

tuner.search(
    x={'input_ids': x['input_ids']},
    y={'issue': y_issue, 'product': y_product},
    epochs=5)
```

### Full error log
Or allmost - removed some 100 lines of `2020-11-04 16:53:37.6569xx: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size xxxxxxxx totalling xxxMiB `

```
2020-11-04 16:53:37.656906: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2405376 totalling 2.29MiB  
2020-11-04 16:53:37.656913: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2457600 totalling 2.34MiB  
2020-11-04 16:53:37.656919: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2950912 totalling 2.81MiB  
2020-11-04 16:53:37.656926: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2967552 totalling 2.83MiB  
2020-11-04 16:53:37.656933: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3091200 totalling 2.95MiB  
2020-11-04 16:53:37.656939: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3104512 totalling 2.96MiB    
2020-11-04 16:53:37.656946: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3161088 totalling 3.01MiB  
2020-11-04 16:53:37.656953: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3555840 totalling 3.39MiB  
2020-11-04 16:53:37.656960: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3575808 totalling 3.41MiB  
2020-11-04 16:53:37.656966: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4012032 totalling 3.83MiB  
2020-11-04 16:53:37.656973: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 18 Chunks of size 4110336 totalling 70.56MiB
2020-11-04 16:53:37.656980: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4111104 totalling 3.92MiB  
2020-11-04 16:53:37.656987: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4207872 totalling 4.01MiB  
2020-11-04 16:53:37.656994: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4236544 totalling 4.04MiB  
2020-11-04 16:53:37.657001: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4245504 totalling 4.05MiB  
2020-11-04 16:53:37.657008: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4354816 totalling 4.15MiB  
2020-11-04 16:53:37.657015: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 809 Chunks of size 9437184 totalling 7.11GiB 
2020-11-04 16:53:37.657023: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 143 Chunks of size 9830400 totalling 1.31GiB 
2020-11-04 16:53:37.657030: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10106624 totalling 9.64MiB   
2020-11-04 16:53:37.657037: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10291200 totalling 9.81MiB   
2020-11-04 16:53:37.657044: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10386432 totalling 9.91MiB   
2020-11-04 16:53:37.657051: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10658560 totalling 10.16MiB  
2020-11-04 16:53:37.657059: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10850048 totalling 10.35MiB  
2020-11-04 16:53:37.657066: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10895872 totalling 10.39MiB  
2020-11-04 16:53:37.657073: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10896384 totalling 10.39MiB
2020-11-04 16:53:37.657080: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 11045120 totalling 10.53MiB                                                                         
2020-11-04 16:53:37.657087: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 11188224 totalling 10.67MiB
2020-11-04 16:53:37.657093: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 11581440 totalling 11.04MiB
2020-11-04 16:53:37.657100: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 19 Chunks of size 11796480 totalling 213.75MiB
2020-11-04 16:53:37.657108: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12137472 totalling 11.58MiB
2020-11-04 16:53:37.657115: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12404736 totalling 11.83MiB
2020-11-04 16:53:37.657122: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12939264 totalling 12.34MiB
2020-11-04 16:53:37.657129: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 13547520 totalling 25.84MiB
2020-11-04 16:53:37.657136: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13621248 totalling 12.99MiB
2020-11-04 16:53:37.657144: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13785856 totalling 13.15MiB
2020-11-04 16:53:37.657151: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13865984 totalling 13.22MiB
2020-11-04 16:53:37.657160: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 14155776 totalling 40.50MiB
2020-11-04 16:53:37.657164: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 14304768 totalling 13.64MiB                                                       
2020-11-04 16:53:37.657168: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 14649600 totalling 13.97MiB                                                                         
2020-11-04 16:53:37.657172: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 35 Chunks of size 15360000 totalling 512.70MiB
2020-11-04 16:53:37.657176: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 15906816 totalling 45.51MiB
2020-11-04 16:53:37.657181: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 17 Chunks of size 16515072 totalling 267.75MiB
2020-11-04 16:53:37.657186: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 16575488 totalling 15.81MiB
2020-11-04 16:53:37.657194: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 17123328 totalling 16.33MiB
2020-11-04 16:53:37.657198: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 17658112 totalling 16.84MiB
2020-11-04 16:53:37.657202: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18265088 totalling 17.42MiB
2020-11-04 16:53:37.657206: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18265856 totalling 17.42MiB
2020-11-04 16:53:37.657211: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 31 Chunks of size 18266112 totalling 540.02MiB
2020-11-04 16:53:37.657215: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 18266368 totalling 34.84MiB
2020-11-04 16:53:37.657219: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18315776 totalling 17.47MiB
2020-11-04 16:53:37.657223: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18501888 totalling 17.64MiB
2020-11-04 16:53:37.657229: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18659328 totalling 17.79MiB
2020-11-04 16:53:37.657234: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18659584 totalling 17.79MiB
2020-11-04 16:53:37.657240: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 24899584 totalling 23.75MiB
2020-11-04 16:53:37.657245: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 44 Chunks of size 39321600 totalling 1.61GiB
2020-11-04 16:53:37.657248: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 28 Chunks of size 93763584 totalling 2.44GiB
2020-11-04 16:53:37.657252: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 95564544 totalling 91.14MiB
2020-11-04 16:53:37.657256: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 99225600 totalling 94.63MiB
2020-11-04 16:53:37.657260: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 102592512 totalling 97.84MiB
2020-11-04 16:53:37.657264: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 103200768 totalling 98.42MiB
2020-11-04 16:53:37.657268: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 122075136 totalling 116.42MiB
2020-11-04 16:53:37.657273: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 143917056 totalling 137.25MiB
2020-11-04 16:53:37.657278: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 173402880 totalling 165.37MiB
2020-11-04 16:53:37.657283: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 174149120 totalling 166.08MiB
2020-11-04 16:53:37.657287: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 184383744 totalling 175.84MiB
2020-11-04 16:53:37.657291: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 184633344 totalling 176.08MiB
2020-11-04 16:53:37.657295: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 19.95GiB
2020-11-04 16:53:37.657299: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_: 21468487680 memory_limit_: 21468487808 available bytes: 128 curr_region_allocation_byt
es_: 42936975872
2020-11-04 16:53:37.657313: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats:
Limit:                     21468487808
InUse:                     21422771968
MaxInUse:                  21432602368
NumAllocs:                    46783782MaxAllocSize:                187526912Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2020-11-04 16:53:37.657683: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ****************************************************************************************************
2020-11-04 16:53:37.657727: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at matmul_op.cc:481 : Resource exhausted: OOM when allocating tensor with shape[3200,3072] and type float o
n /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "myLib/python3.7/site-packages/kerastuner/engine/base_tuner.py", line 131, in search
    self.run_trial(trial, *fit_args, **fit_kwargs)
  File "myLib/python3.7/site-packages/kerastuner/engine/multi_execution_tuner.py", line 98, in run_trial
    history = model.fit(*fit_args, **copied_fit_kwargs)
  File "myLib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "myLib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit
    tmp_logs = train_function(iterator)
  File "myLib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "myLib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 840, in _call
    return self._stateless_fn(*args, **kwds)
  File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
    cancellation_manager=cancellation_manager)
  File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 550, in call
    ctx=ctx)
  File "myLib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted:  OOM when allocating tensor with shape[3200,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc         [[node BERT_MultiLabel_MultiClass/tf_bert_model/bert/encoder/layer_._11/intermediate/dense/Tensordot/MatMul (defined at myLib/python3.7/site-packages/transformers/modeling_tf_bert.py:327) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[clip_by_norm_1/truediv/_546]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted:  OOM when allocating tensor with shape[3200,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc         [[node BERT_MultiLabel_MultiClass/tf_bert_model/bert/encoder/layer_._11/intermediate/dense/Tensordot/MatMul (defined at myLib/python3.7/site-packages/transformers/modeling_tf_bert.py:327) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_1394299]

Function call stack:
train_function -> train_function
```
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
