Hight CPU and low GPU on XLNet
## 🐛 Bug

I am running Bert, GPT, GPT2, XLNET. I got very high CPU usage (e.g. 16 cores) with  XLNet while the others (Bert, GPT, GPT2) dont.

For BERT, GPT, GPT2: CPU 1 cores, 100%GPU 
For XLNet: CPU 16 cores, 50 to 60% GPU 


Is there any hidden implementation which requires CPU?

<!-- Important information -->

Model I am using (XLNet....): XLNET

Language I am using the model on: English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu 18.04
* Python version: 3.6
* PyTorch version: 1.0
* PyTorch Transformers version (or branch): pytorch_transformers 1.2.0
* Using GPU ? RTX 2080 TI
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

I also meet the above problem with XLNet. The GPU usage is very low and unstable, but the CPU usage is very high. The running speed is very low.

Are there any ops running on CPU rather than GPU in your XLNet implementation? How to improve the GPU usage and speed up the running speed ? Thanks!

Environment:
Pytorch: 1.1.0
GPU: V100, 16G
pytorch_transformers: 1.2.0
OS: centos 7.6
Python: 3.6<|||||>Does it occur while training or predicting? Are you sure your gpu is available (to PyTorch or TensorFlow)? What do logs say? <|||||>It is likely that this is caused by the tokenisation rather than the model training. Tokenisation typically happens on the CPU, the token_ids are then transferred to the GPU and from then on out the training happens on the GPU. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>The same problem. How to solve it?