XLNet is getting slower when enabling mems
Per API doc, using mems help to reduce inference time. However, I noticed that more time is needed when increasing mem_len. Do I misunderstand the usage of mem_len parameter? 
Here is the testing code

```
import time
import torch
from transformers import XLNetTokenizer, XLNetLMHeadModel

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')

text = """
    A horrible, messy split second presents
    itself to the heart-shaped version as Scott is moved. The upcoming movie benefits at 
    the mental cost of ages 14 to 12. Nothing substantial is happened for almost 48 days. 
    When that happens, we lose our heart. <eod> The quick brown fox jumps over the lazy dog. <mask>
"""

mems = None

input_ids = tokenizer.encode(text)
input_ids = torch.tensor(input_ids).unsqueeze(0)

epoch = 20

for men_len in [0, 16, 32, 64, 128]:
    model = XLNetLMHeadModel.from_pretrained('xlnet-base-cased', mem_len=men_len)

    start_dt = time.monotonic()
    for i in range(epoch):
        outputs = model(input_ids=input_ids, mems=mems)
        if men_len > 0:
            mems = outputs[1]
    end_dt = time.monotonic()

    print('Average Duration for men_len {}: {}'.format(men_len, round(end_dt-start_dt, 2)))
```

Output is 
```
Average Duration for men_len 0: 2.49
Average Duration for men_len 16: 2.62
Average Duration for men_len 32: 2.67
Average Duration for men_len 64: 2.81
Average Duration for men_len 128: 3.28
```
When using `past` or `mems` values, you should be careful not to give the model the input ids which have already been computed. We've recently [added a documentation section ](https://huggingface.co/transformers/quickstart.html#using-the-past)detailing the use of `past`, which is similar to the way `mems` should be used. 

Please notice we're only feeding the model the tokens for which the attention values have not be computed yet; which is only the last token in the case of sequential decoding.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
