Using Transformer Library for code prediction
Dear all,

I am new in exploring transformer library. I would like to use transformer models to train on my own text corpus (.txt) containing C++ source code tokens seperated with space characters. I would like to provide tokenized C++ source code files from multiple repositories in textual format (.txt) , and function should give me trained models with accuracy results, which I can use for code prediction latter on.

I have came accross with [Deep TabNine](https://tabnine.com/blog/deep/), which has used GPT2. But, I donot know about the following:

1. How could I train tranformer's library GPT2 model for C++ tokenized code? 
2. Can I use all transformer models such as BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet etc to train on my own C++ tokenized code? 
3. If no, which one can be used which one cannot and how? 
4. Is it advisable to use transformer pretrained models by fine tunning over my own textual corpus of C++ tokens? or should I build trained models from the scratch by using transformer library?


Please let me know about it.

You should probably train a model from scratch.

Here a few links that are relevant:

- our blog post on [how to train a model from scratch](https://huggingface.co/blog/how-to-train) using `transformers` and `tokenizers`.
- specifically on the topic of code, we just uploaded [CodeBERTa](https://huggingface.co/huggingface/CodeBERTa-small-v1#codeberta), a model pretrained on the `CodeSearchNet` dataset from GitHub (+ fine-tuned to a classification task)

Let us know how it goes.