Using memory states with XLNet / TransfoXL
I would like to fine-tune XLNet / TransfoXL for a classification task where I classify each sentence in the context of a large document. Is there an example for how to use the memory states in XLNet and TransfoXL? 
This example only uses memory states for inference but there is no example for training:
https://github.com/huggingface/pytorch-transformers/blob/xlnet/examples/single_model_scripts/run_transfo_xl.py

This example doesn't use the memory states:
https://github.com/huggingface/pytorch-transformers/blob/24ed0b9346079da741b952c21966fdc2063292e4/examples/run_xlnet_classifier.py

Naively feeding in the memory states leads to some dimension mismatch at the end of the training epoch:
```
  File "/home/lambda/repos/research/trainer/models/xlnet.py", line 86, in forward
    mems=new_mems
  File "/home/lambda/python-envs/research/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lambda/repos/pytorch-transformers/pytorch_transformers/modeling_xlnet.py", line 959, in forward
    new_mems = new_mems + (self.cache_mem(output_h, mems[i]),)
  File "/home/lambda/repos/pytorch-transformers/pytorch_transformers/modeling_xlnet.py", line 792, in cache_mem
    new_mem = torch.cat([prev_mem, curr_out], dim=0)[-self.mem_len:]
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 14 and 32 in dimension 1 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71
```
Which command did you use to "naively" feed in the memory states?
You can just feed the mems that you get from the previous forward pass, but the inputs need to be the continuation of the previous input. So the batch_size, in particular, should stay the same.<|||||>I found this post that talks about how to organize the inputs: https://mlexplained.com/2019/07/04/building-the-transformer-xl-from-scratch/

Is there an example for how to adapt this for classification? Right now the data is organized as `(batch_size x max_seq_length)` and labels are `batch_size`. Each example in the batch represents  a sentence, where multiple sentences may come from the same document. For simplicity we can assume they are all from the same document.<|||||>I was looking at doing the same with TransformerXL, but ran into this same issue regarding how to adapt the label vector to work with the data matrix when trying to do classification. I'd appreciate any help from people that have successfully implemented this.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
