Time for second encoding is much higher than first time
Hi,

using a bert model on a single gpu  to encode multiple times after each each other like 
```
bert_model = TFBertModel.from_pretrained('bert-base-cased',output_hidden_states=True)

input = tokenizer(data , max_length=MAX_SEQ_lEN,padding="max_length",truncation=True, return_tensors="tf")
outputs1 = bert_model(input)
###time1 : 0.1 seconds
outputs2 = bert_model(input)
### time2: 1.7 seconds

```
gives a unproportional high time for the second encoding. If the first encoding time is just 0.1 second I would assume to have each other encoding afterweards also about 0.1 seconds. I run this multiple times and it seems a patterns that the encoding after the first encoding is significantly larger.

Can someone explain this behaviour? I assume it is due to gpu. 

```
Env: win 10, python: 3.6, tensorflow 2.3, transformers 3.3.1
GPU: Nvidia mx 150
```

Could you show the entire code, including how you instantiate the model, as well as your environment information, as mentioned in the template? Thank you.<|||||>that's ok?<|||||>With the following code:

```py
from transformers import TFBertModel, BertTokenizer
from time import time
import tensorflow as tf

print("GPU Available:", tf.test.is_gpu_available())

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
bert_model = TFBertModel.from_pretrained('bert-base-cased',output_hidden_states=True)

input = tokenizer("Hey is this slow?" * 100 , max_length=512,padding="max_length",truncation=True, return_tensors="tf")

for i in range(100):
    start = time()
    outputs = bert_model(input)
    print(time() - start)
```

Running on CPU doesn't increase the time for me:

```
GPU Available: False
0.5066382884979248
0.5038580894470215
0.5125613212585449
0.5018391609191895
0.4927494525909424
0.5066125392913818
0.49803781509399414
0.5140326023101807
0.501518726348877
0.49771928787231445
0.5038976669311523
```

[Running on GPU, no problem either:](https://colab.research.google.com/drive/1tfynzpOiQJKkEi0vkpKaDXTwhB5-k6Td?usp=sharing)

```
GPU Available: True
0.09349918365478516
0.09653115272521973
0.09893131256103516
0.10591268539428711
0.09297466278076172
0.09105610847473145
0.10088920593261719
0.0935661792755127
0.09639692306518555
0.10130929946899414
0.0947415828704834
0.09380221366882324
```<|||||>Thanks. If I do exactly your code I observe an increasing time! Very strange, but I assume that this has something to do with the gpu memory not releasing?

```
0.07779383659362793
0.20029330253601074
0.2085282802581787
0.22140789031982422
0.23041844367980957
0.22839117050170898
0.23337340354919434
0.22336935997009277
0.22971582412719727
0.22768259048461914
0.22839140892028809
0.22934865951538086
0.23038363456726074
0.22646212577819824
0.23062443733215332
0.22713351249694824
0.24032235145568848
0.24936795234680176
0.24984216690063477
0.2523007392883301
0.2481672763824463
0.2532966136932373
0.24833273887634277
0.2513241767883301
0.2522923946380615
0.2536492347717285
0.25013017654418945
0.25212621688842773
0.24585843086242676
0.25535058975219727
0.2563152313232422
0.2423419952392578
0.6144394874572754
0.647824764251709
0.6494302749633789
0.6406776905059814
0.6507377624511719
0.6411724090576172
0.6513652801513672
0.6484384536743164
0.6489207744598389
0.6405856609344482
0.6493120193481445
0.6484384536743164
0.6372919082641602
0.6494011878967285
0.6433298587799072
0.65077805519104
0.6475985050201416
0.6383304595947266
0.6525297164916992
0.6413178443908691
0.6475212574005127
0.6485188007354736
0.64430832862854
0.6478779315948486
0.6457436084747314
0.7288320064544678
0.6573460102081299
0.6572368144989014
0.5861053466796875
0.6324939727783203
0.722456693649292
0.6353938579559326
0.6324222087860107
0.6373186111450195
0.6216456890106201
0.6627655029296875
0.7275354862213135
0.6035926342010498
0.6590445041656494
0.5936176776885986
0.6416335105895996
0.6400752067565918
1.1317992210388184
1.2438006401062012
1.2430295944213867
1.2435650825500488
1.2585129737854004
1.2704930305480957
1.2204067707061768
1.2424969673156738
1.2366819381713867
1.2533769607543945
1.2510595321655273
1.2426464557647705
1.2566087245941162
1.2392685413360596
```

If you don't mind, my issue regarding the usage for the input of the tokenizer is still open. :)
https://github.com/huggingface/transformers/issues/7674<|||||>Indeed, I'll try and check the issue ASAP. Thanks for the reminder!<|||||>@LysandreJik . Thank you! But, do you have any idea for my issue? It seems it is a gpu issue?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.