Implementing XLNet in pytorch
Due to the new work of [XLNet](arxiv.org/abs/1906.08237 ) and the implementation of it in tensorflow maybe we need to add this one to the current repository.

https://github.com/zihangdai/xlnet
FYI @roholazandie we are currently working on XLNet with pytorch over here
https://github.com/pingpong-ai/XLNet-pytorch/tree/dev/poc<|||||>I'll add it here also. I was working on a coming release this week anyway.
It's a mix of BERT/Transformer-XL and something I was also playing with (Two-Stream Self-Attention) so hopefully, it won't delay too much the release. <|||||>Awesome, downstream libraries like [flair](https://github.com/zalandoresearch/flair) are really looking forward to use XLNet 🤗 (So expect results on CoNLL and PoS tagging whenever XLNet is implemented here)<|||||>and [jiant](https://jiant.info)!<|||||>I had finished Simple XLNet implementation with Pytorch Wrapper here : https://github.com/graykode/xlnet-Pytorch<|||||>How is this project going?

I'm debating whether to try to build XLNet into jiant directly, but I'm not eager to replicate your hard work. Tx!<|||||>Yes we're on track to finish the release this week I think (or next Monday in the worse case).

We reproduced the results of XLNet on STS-B (Pearson R > 0.918), the GLUE task showcased on the TF repo, with the same hyper-parameters (didn't try the others tasks but the model is the same for all).

It's taking a bit more time than planned because we took the opportunity to refactor the library's back-end. I'm really excited about the new release.

We will now have 6 different architectures (BERT, GPT, GPT-2, Transformer-XL, XLNet and XLM) and over 25 associated pretrained weights, all with the same API (for instance the GLUE training script is now the same for all the models!) and direct access to all the models' internals.

And a lot of other things (a lot of tests to avoid future regressions, compatibility with TorchScript, easy serialization and loading of fine-tuned models...)<|||||>> I'll add it here also. I was working on a coming release this week anyway.
> It's a mix of BERT/Transformer-XL and something I was also playing with (Two-Stream Self-Attention) so hopefully, it won't delay too much the release.

@thomwolf , very much looking forward to your implementation of BERT/Transformer-XL. Wondering if you were planning to release that too, and if so where you are planning to make it available. Thanks so much <|||||>Is there any direct wrapper for QA task using XLNet?<|||||>I appriciate your hard work. I just saw it today, and you almost implemented.

Is it really better than Bert, which i find to be amazing so far?<|||||>Under the conditions in their paper, yes. Question is whether it always is: in what tasks, using the same data set; with same compute time (or energy consumption) for training (+on what hardware as same speed on hardware X doesn't imply same speed on hardware Y); with same number of parameters (model size); with architecture size for same prediction speed; consistently / on average (+confidence interval) for repeated runs with random initialisation...