BUG for XLNet: Low GPU usage and High CPU usage, very low running speed!
## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLNet

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: XLNet used as encoder for seq2seq generation 

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Centos 7.6
* Python version: 3.6
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.2.0
* Using GPU ? V100, 16G
* Distributed of parallel setup ? Use multiprocessing for multi GPU training
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

Hi, could you provide a minimal code that reproduces the issue so that we may help debug/check on our side?<|||||>hey @LysandreJik,

I think I am experiencing the same/similar issue. Below are the sample code, a screenshot showing the high CPU and low GPU usage, and the `requirements.txt` file.

In terms of technical setup, I used a docker container deployed with [vast.ai](https://vast.ai/) with RTX 2070S, CUDA 10.1, AMD Ryzen 5 2600X (6c/12t), 16GB. It used the [latest PyTorch image](https://hub.docker.com/r/pytorch/pytorch) as of the time of writing. From my observations the fluctuating CPU usage seems to max out at 600%, hinting at the 6 physical cores. However. I think I have experienced the issue with 2080 Ti and 1080 Ti, as well.

Thank you!

```python
# -*- coding: utf-8 -*-

import torch
from transformers import XLNetTokenizer, XLNetLMHeadModel
import requests

convai1_data = requests.get('http://convai.io/2017/data/train_full.json').json()

for dial in convai1_data:
    utterances = [thread_line['text'] for thread_line in dial['thread']]
    dial['utterances'] = utterances
    dial['predictions'] = dict()

convai1_data = [dial for dial in convai1_data if len(dial['utterances']) > 2]

# https://github.com/huggingface/transformers/issues/917#issuecomment-525297746
def xlnet_sent_probability(PADDING_TEXT, text):
    tokenize_text = model_tokenizer.tokenize(text)[:512]
    tokenize_input = model_tokenizer.tokenize(PADDING_TEXT)[:511] + ['<eod>'] + tokenize_text

    sentence_word_probs = list()
    sentence_best_word_probs = list()
    best_words = list()

    for max_word_id in range((len(tokenize_input)-len(tokenize_text)), (len(tokenize_input))):

        sent = tokenize_input[:]

        input_ids = torch.tensor([model_tokenizer.convert_tokens_to_ids(sent)])

        perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
        perm_mask[:, :, max_word_id:] = 1.0 

        target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float)
        target_mapping[0, 0, max_word_id] = 1.0

        if torch.cuda.is_available():
            input_ids = input_ids.cuda()
            perm_mask = perm_mask.cuda()
            target_mapping = target_mapping.cuda()

        with torch.no_grad():
            outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping)
            next_token_logits = outputs[0]  # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size]
            predicted_prob = torch.softmax(next_token_logits[0][-1], dim=-1)

        predicted_prob = predicted_prob.detach().cpu().numpy()

        word_id = model_tokenizer.convert_tokens_to_ids([tokenize_input[max_word_id]])[0]
        word_prob = predicted_prob[word_id].item()

        sentence_word_probs.append(word_prob)

    return sentence_word_probs

for XLNET_MODEL in ['xlnet-base-cased', 'xlnet-large-cased']:

    model_tokenizer = XLNetTokenizer.from_pretrained(XLNET_MODEL)
    model = XLNetLMHeadModel.from_pretrained(XLNET_MODEL)

    if torch.cuda.is_available():
        model = model.cuda()

    model = model.eval()

    for dial in convai1_data:
        utterances = dial['utterances']

        sentences_word_probs = list()

        for u1, u2 in zip(utterances[:-1], utterances[1:]):
            sentence_word_probs = xlnet_sent_probability(u1, u2)

            sentences_word_probs.append(sentence_word_probs)

        dial['predictions'][XLNET_MODEL+'_sentences_word_probs'] = sentences_word_probs

```

[requirements.txt](https://github.com/huggingface/transformers/files/3883834/requirements.txt)
[cpu_and_gpu_usage](https://user-images.githubusercontent.com/8143425/69497875-e9226f80-0ee1-11ea-8098-f81df3972f3e.png)



<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>The same problem. How to solve it?