some words not in xlnet vocabulary ,especially name 
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I face the problem when xlnet tokenizer encodes name 

    from pytorch_transformers import *
    import torch
    tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased') 
    model = XLNetModel.from_pretrained('xlnet-base-cased')
    sents = ["here is a dog","michael love mary <pad>"]
    input_ids =[tokenizer.encode(sent) for sent in sents]

    model(torch.tensor(input_ids))[0]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: expected sequence of length 4 at dim 1 (got 8)

I think the reason is that xlnet can't encode the name "michael " and "mary" , instead it encodes "michael" to "_mi" "cha" "el" and mary  to "m" "ary"  respectively . And this result leads input_ids to has two different length id list . As the result , when I fed the torch.tensor(input_ids) to XLNetModel would cause "ValueError" problem , how could I fix it?

    input_ids
    [[193, 27, 24, 2288], [12430, 2409, 530, 564, 17, 98, 1449, 5]]
    [tokenizer.tokenize(sent) for sent in sents]
    [['▁here', '▁is', '▁a', '▁dog'], ['▁mi', 'cha', 'el', '▁love', '▁', 'm', 'ary', '<pad>']]

Hi!

XLNet uses a SentencePiece tokenizer which splits the words into subword units. In your case, it splits the two examples in different sized sequences, which can't be parsed as a Tensor which requires an input matrix (and not a list of lists).

You should pad your sequences after they have been tokenized so that they all are of equal size. Then they can be converted to a tensor and fed to the model.<|||||>> Hi!
> 
> XLNet uses a SentencePiece tokenizer which splits the words into subword units. In your case, it splits the two examples in different sized sequences, which can't be parsed as a Tensor which requires an input matrix (and not a list of lists).
> 
> You should pad your sequences after they have been tokenized so that they all are of equal size. Then they can be converted to a tensor and fed to the model.

Thanks for your reply.
My purpose is to fed each word's contextual embedding to transformer layer to obtain sentence embedding
However ,  by this way I can't get each word's contextual embedding , is there any solution to fix this problem or just use xlnet function to get sentence embedding?<|||||>If you're looking to create sentence embeddings based on transformers, I'd like to redirect you to [this issue](https://github.com/huggingface/pytorch-transformers/issues/876) that discusses exactly this.

It discusses the use of the [UKPLab sentence-transformers library](https://github.com/UKPLab/sentence-transformers) which is built on top of our library and which can provide sentence embeddings based on XLNet.