Fine tune GPT-2 pytorch
Hello,

I want to fine tune GPT-2 (PyTorch version) on a custom dataset. Words or small phrases of the dataset are marked, for example:

_some text [ss] word / small phrase [se] some other text._

I want to generate this kind of text with GPT-2, so firstly I thought to add [ss] and [se] as special tokens.

I am looking for a training script sample for GPT-2, to see how to prepare data as input for the model (if preprocessing, or specific format is needed), which type of loss to use, etc. but I cannot find any. I also looked through the [library](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py) and also didn't find the training script.

Are there any suggestions?

Thank you in advance.
P.s. if this is not the appropriate place for this question, feel free to direct me accordingly.
This repo(https://github.com/kifish/GPT4NLG) will help you out.<|||||>Have you taken a look at the [`run_clm.py` script](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py) in the examples? It seems to be doing exactly what you're looking for.

Erratum: I just realized that your link to `library` was actually a link to `run_clm.py`. What do you mean you didn't find the training script? Do you mean to say you would like an understandable guide on how to fine-tune GPT-2? Then [this guide](https://github.com/ncoop57/i-am-a-nerd/blob/master/_notebooks/2020-05-12-chatbot-part-1.ipynb) on DialoGPT, which is based off of the GPT-2 architecture, may be helpful.<|||||>@kifish thank you for your reply, it is an interesting repository.

I am doing this for the first time so I am looking for something more simple, although I got some ideas reading the code which I will use.<|||||>@LysandreJik thank you , for the helpful tutorial.

When I mentioned that I didn't find the training script, I meant that in line 327 of the `run_clm.py`, the `main` method calls the `train` method of a `Trainer`, but I couldn't find the code of the `train` method.

In addition, for the data preparation as input for the model, except for the extra tokens that I mentioned in my initial post, I thought to also add a `bos` and `eos` token at the begging and at the end of each text respectively, so that the model learns when a text starts and ends. The GPT-2 Tokenizer has already these tokens but they have the same id, they both have the id 50256. What is the reason behind this? 
In order to deal with this, another way to prepare data is to use just the `eos` token to denote the end of a text, since the model should basically learn when a text ends. Can you please explain briefly?

Thank you in advance.

<|||||>The code for the `train` method is in the `Trainer` class that you can find [here](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L565). It's a bit encapsulated so it might be a bit hard to follow - but we're working on simpler examples that show a basic PyTorch training loop and which should be out in a few months.

You may also find [this notebook interesting, which goes into finer detail on how to train a language model.](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb)

Regarding your question on the extra tokens, adding a `bos` and `eos` token depends on how the model was pre-trained. BERT requires these as they were used during its pre-training. However, GPT-2 has very few special tokens during pre-training: a single `<|endoftext|>` token that was placed between sequences. I recommend you read the GPT-2 paper to get an idea of their pre-processing; we try to stay as close to the original implementation as possible.<|||||>@LysandreJik thank you for your reply.

I am writing the code for training GPT-2 and firstly, as you suggested, I concatenated the input texts separated by `<|endoftext|>` token and then split it into fixed lengths, which is the model input.

In the GPT-2 casual language modelling, the model input is also the labels of the model, so to compute the loss, the input and labels arguments of the loss function are the same. The [tutorial](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) that you mentioned in the previous post, when references the model input and labels talks about a shifting on the left or on the right.

To calculate the loss correctly, are the model input and labels arguments of the loss function the same, or am I missing something?<|||||>If you pass the labels to the model, you should pass exactly the same value as the input IDs. The model then shifts the label IDs and calculates the loss on its own.<|||||>@LysandreJik I have a question regarding training. 

For a Language Model which are indicative metrics to monitor training? Because for now I just consider the loss. For example, in classification it is usually considered the F1 score.
Additionally, in the tutorial that you shared, no early stopping was used in training, and the model was trained for three epochs. I have a small dataset of ~8K small texts (2MB). Do you have any suggestions on how to train the model, i.e. if there is no early stopping, how to decide when to stop training and how can I evaluate training?

You have been very helpful,
regards.<|||||>Hello @kifish 

the GPT4NLG repo that you have shared back then was very helpful but I am no longer able to see it. Can you do anything about that?

Thank you in advance.<|||||>> Hello @kifish
> 
> the GPT4NLG repo that you have shared back then was very helpful but I am no longer able to see it. Can you do anything about that?
> 
> Thank you in advance.

https://github.com/kifish/GPT4NLG/tree/github<|||||> https://github.com/kifish/GPT4NLG/tree/github

Thank you @kifish. If possible, let it visible for some days.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.