How to set the token_type_ids in XLNet correctly?
Hi, I am fine-tuning an XLNet, and I want to use type embedding to indicate different parts of a sequence. I am facing a difficulty that `“indices are selected in the vocabulary (unlike BERT which has a specific vocabulary for segment indices)”` which is the description of `token_type_ids` in the official document. Does that mean the type embedding and token embedding share the same vocabulary? In that case, how can I select the right indices for the types?  If I use `0` and `1` for types, is there a collision between types and the special tokens (like `UNK`)?

Thanks in advance!

Hello! We have an example using `token_type_ids` in our `run_glue` script. You can look at how we build the features in the  [`utils_glue`, especially concerning the `segment_ids`](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_glue.py#L456-L484) which are the `token_type_ids` that will be fed to the model. 

If I recall correctly the XLNet model has `0` for the first sequence `token_type_ids`, `1` for the second sequence, and `2` for the last (cls) token.
<|||||>> Hello! We have an example using `token_type_ids` in our `run_glue` script. You can look at how we build the features in the [`utils_glue`, especially concerning the `segment_ids`](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_glue.py#L456-L484) which are the `token_type_ids` that will be fed to the model.
> 
> If I recall correctly the XLNet model has `0` for the first sequence `token_type_ids`, `1` for the second sequence, and `2` for the last (cls) token.

Thank you for your explanation!

I still have a question, the code in `utils_glue` is for BERT. As far as I know, the token embeddings and type embeddings are selected from two embedding matrices in BERT, therefore, the type index `0` won't give you a `PAD` token embedding. In XLNet, the type indices are selected in the vocabulary, in which `0` index represents `UNK` token and `1` index represents `BOS` token. 

Do I misunderstand the meaning of `"indices are selected in the vocabulary"`, or we can freely use the `BOS` `EOP` `EOD` tokens for our type embeddings?
<|||||>Oh, that's a typo in XLNet docstring that I thought we had corrected already.
Thanks for reminding us of that.

The type indices in XLNet are not selected in the vocabulary, they can be arbitrary.
In XLNet segment ids (what we call `token_type_ids in the repo) don't correspond to embeddings, they are just numbers and the only important thing is that they have to be different for tokens which belong to different segments, hence the flexibility in the exact values (XLNet is using relative segment difference with just two segment embeddings: 0 if the segment id of two tokens are the same, 1 if not). See [here](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_xlnet.py#L926-L928).