Conversion of gpt-2 small model
Hey! This seems like something a lot of folks will want. I'd like to be able to load GPT-2 117M and fine-tune it. What's necessary to convert it? I looked at the tensorflow code a little and it looks vaguely related to transformer xl, but I haven't looked at the paper yet or etc.
I'd like to help out on this. I will have a look and try to understand the earlier bridges in this repo. Let me know if you see anywhere a newcomer can be helpful with.<|||||>Sure, would be happy to welcome a PR.
You can start from `modeling_openai.py` and `tokenization_openai.py`'s codes.
It's pretty much the same architecture (read OpenAI's paper first!)
You should mostly reimplement the BPE tokenization to work byte-level and move the layer norms modules to the input rather than the output of the layers.<|||||>Ok, GPT-2 should be in the coming 0.6.0 release (see #287)<|||||>Ok it's on pip: https://github.com/huggingface/pytorch-pretrained-BERT/releases/tag/v0.6.0
Please read the updated README for details. All should be there (model and examples).
Have a nice week y'all.<|||||>@thomwolf do we have pytorch implementation of GPT-2 small? <|||||>Yes, just read the README