XLnet memory usage for long sequences
## ❓ Questions & Help

Hello,
I have some questions regarding how the XLnet memory and output work in this implementation.

1. As it's been mentioned before, by default, XLnet doesn't use memory. So, how is this possible that it accepts long sequences as input (in other words, why there isn't any limit on the number of input tokens), unlike BERT, for example, that will only accept 512 tokens.

2. If I set the memory length to 512, and feed the XLnet with 512 tokens at a time (for a 1024 sequence length), also pass the memory in each step. (like the example code below) Will the final output of the network, includes all the information from the whole sequence?

```
tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')
model = XLNetModel.from_pretrained('xlnet-large-cased')
mems = None
for i in range(2):
    input_ids = torch.tensor(tokenizer.encode(text[i])).unsqueeze(0)
    outputs = model(input_ids, mems=mems)
    mems = outputs[1]
```

To be clear, I want to use XLnet for long text summarization. So I need to feed the XLnet output to a decoder part and need a fixed-length representation.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
