Get hidden states from all layers of Transformer-XL?
Hi,

Thank you for supporting the pretrained Transformer-XL model! I was wondering if it makes sense to get hidden states from all layers of Transformer-XL as the output, just as what can be done for BERT. It seems this is not supported currently. Practically I found this strategy worked well for BERT and gave better results. Not sure if it is a good idea for Transformer-XL. Thank you!
Hi @hugochan, actually that what's in the `mems` of the Transformer-XL are (maybe you can read again the paper).

One thing to be careful about is that the `mems` have transposed first dimensions and are longer (see the readme). Here is how to extract the hidden states from the model output:
```python
hidden_states, mems = model(tokens_tensor)
seq_length = hidden_states.size(1)
lower_hidden_states = list(t[-seq_length:, ...].transpose(0, 1) for t in mems)
all_hidden_states = lower_hidden_states + [hidden_states]
```<|||||>Hi @thomwolf , thank you for your answer! Just one quick question. It seems that `mems` already contains a list of num_layer hidden states, what is the difference between `lower_hidden_states[-1]` and `hidden_states` in your code? Thank you!<|||||>Actually `mems` contains all the hidden states PLUS the output of the embeddings (`lower_hidden_states[0]`) so `lower_hidden_states[-1]` is the output of the hidden state of the layer below the last layer and `hidden_states` is the output of the last layer (before the softmax).

I will add a note on that in the readme.