bidirectional training for GPT2
# 🚀 Feature request

May I ask, if there is possible to fine-tune GPT2 in bidirectional training and bidirectional training together? 
Like the UniLM: https://github.com/microsoft/unilm/tree/master/unilm-v1


## Motivation

GPT2 is widely used in text generation in unidirectional training (left-to-right). Could we use GPT2 to process context in bidirectional, while generating text in unidirectional. 

Thanks for any feedback and really appreciate for any tips.
Hi,

I've answered your question on the [forum](https://discuss.huggingface.co/t/fine-tuning-gpt2-for-bidirectional-training/12155/4).<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.