Extract Features for GPT2 and Transformer-XL
Hi everyone,

I'm interested in extracting token-level embeddings from the pre-trained GPT2 and Transformer-XL models and noticed that extract_features.py seems to be specific to BERT.

Can you let us know if you have any plans to provide a similar implementation for models other than BERT?

Alternatively, could you possibly provide some hints for us to extract the token-level embeddings the code you already made available with the models?

Many thanks, great work!
Hi Dan,
You can extract all the hidden-states of Transformer-XL using the snippet indicated in the readme [here](https://github.com/huggingface/pytorch-pretrained-BERT#12-transfoxlmodel).
For the GPT-2 it's not possible right now.
I can add it in the next release (or you can submit a PR).<|||||>@thomwolf  Can we extract final hidden layer representations from GPT-2 1.5 billion models now?