Issues in Training GPT-2 Model from Scratch (Text Generation-Identifying Epoch Value-Perplexity Calculation)
Dear all,

I have trained a GPT-2 model from scratch by following a tutorial mentioned at this [link](https://huggingface.co/blog/how-to-train).

I am mentioning following important code snippets:

```
from pathlib import Path
from tokenizers import ByteLevelBPETokenizer
paths = [str(x) for x in Path(".").glob("**/*.txt")]

# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer()
tokenizer.train(files=paths, vocab_size=50257)
tokenizer.save("/content/drive/My Drive/Model")
```

```
from tokenizers.implementations import ByteLevelBPETokenizer
tokenizer = ByteLevelBPETokenizer(
    "/content/drive/My Drive/Model/vocab.json",
    "/content/drive/My Drive/Model/merges.txt",
)
```

```
import json
config = {
  "_num_labels": 2,
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "do_sample": False,
  "early_stopping": False,
  "embd_pdrop": 0.1,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "is_decoder": False,
  "is_encoder_decoder": False,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_epsilon": 1e-05,
  "length_penalty": 1.0,
  "max_length": 20,
  "min_length": 0,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "no_repeat_ngram_size": 0,
  "num_beams": 1,
  "num_return_sequences": 1,
  "output_attentions": False,
  "output_hidden_states": False,
  "output_past": True,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "resid_pdrop": 0.1,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": True,
  "summary_type": "cls_index",
  "summary_use_proj": True,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": False,
  "use_bfloat16": False,
  "vocab_size": 50257
}
with open("/content/drive/My Drive/Model/config.json", 'w') as fp:
    json.dump(config, fp)

tokenizer_config = {
"max_len": 1024
}
with open("/content/drive/My Drive/Model/tokenizer_config.json", 'w') as fp:
    json.dump(tokenizer_config, fp)
```

Afterwards, I train a model from scratch by using following command:

```
!python run_language_modeling.py \
        --train_data_file='/content/drive/My Drive/Dataset/train.txt' \
        --output_dir='/content/drive/My Drive/Model/v1' \
        --model_type=gpt2 \
        --config_name='/content/drive/My Drive/Model' \
        --tokenizer_name='/content/drive/My Drive/Model' \
        --do_train \
        --num_train_epochs=3 \
        --evaluate_during_training \
        --per_gpu_train_batch_size=2 \
        --eval_data_file='/content/drive/My Drive/Dataset/valid.txt' \
        --do_eval \
        --eval_all_checkpoints \
        --per_gpu_eval_batch_size=2 \
        --block_size=128 \
        --gradient_accumulation_steps=5
```

Model is trained with **good perplexity** of around 4.  After reaching at 55-K steps, learning rate approach to 0 and loss is approximately 1.3. But I do not know how many epoch has been run till that time, because Colab halts the process due to its limitation.

However, I am facing following **issues**:

1. I am using following code to perform text generation, but it does not give me meaningfull generated samples. But, if I use the model, which is **finetuned** on the GPT-2 small by using [scripts](https://github.com/huggingface/transformers/tree/master/examples#language-model-training). That one gives me reasonable generated samples. 
**Am I doing something wrong in generating sample from a model, which is trained from the scratch or there is a need to train a model more or there is a problem in tokenizer train code?** 

```
from transformers import (GPT2LMHeadModel,GPT2Tokenizer,GPT2Config)
model_class, tokenizer_class=GPT2LMHeadModel, GPT2Tokenizer
tokenizer = tokenizer_class.from_pretrained('/content/drive/My Drive/Model/v1')

config = GPT2Config.from_pretrained('/content/drive/My Drive/Model/v1')
model = GPT2LMHeadModel.from_pretrained('/content/drive/My Drive/Model/v1', config=config)
model.to('cuda')

prompt_text = 'hello world'
encoded_prompt = tokenizer.encode(prompt_text, return_tensors="pt")
encoded_prompt = encoded_prompt.to('cuda')

output_sequences = model.generate(
        input_ids=encoded_prompt,
        max_length=400+ len(encoded_prompt[0]),
        do_sample=True,
        num_return_sequences=3,
        top_p=0.9)

generated_sequences = []
for generated_sequence_idx, generated_sequence in enumerate(output_sequences):
        print("=== GENERATED SEQUENCE {} ===".format(generated_sequence_idx + 1))
        generated_sequence = generated_sequence.tolist()

        # Decode text
        text = tokenizer.decode(generated_sequence, clean_up_tokenization_spaces=True)

        # Remove all text after the stop token
        text = text[: text.find("</s>") if "</s>" else None]

        # Add the prompt at the beginning of the sequence. Remove the excess text that was used for pre-processing
        total_sequence = (
            prompt_text + text[len(tokenizer.decode(encoded_prompt[0], clean_up_tokenization_spaces=True)) :]
        )

        generated_sequences.append(total_sequence)
        print(total_sequence)
```

2. Secondly, I am using Colab for experimentation. Due to its [limitation](https://stackoverflow.com/questions/55050988/can-i-run-a-google-colab-free-edition-script-and-then-shutdown-my-computer), my experiments halted two times during language modeling. So, I use "should_continue" flag to continue my language modeling process from where it stops. So, I donot have an idea how many epoch has been run out of 3? Colab only gives last 5000 lines of the output. Up-till now, around 55-K steps has been run. Is there a way to **identify how much epoch has been run by considering these 55-K steps**?

3. I am wondering about how do I get such a good perplexity of around 4 on my validation set. is this because of not using padding? or what can be a possible reason?

Kindly let me know about these concerns.
Maybe your dataset is too small to make much sense even if you get a smaller perplexity.<|||||>Thanks for your response.

Here are the sizes of my corpus:

1. Training set ~ 77 MB
2. Validation set ~ 10 MB
3. Testing set ~ 10 MB

I build this corpus as under:

1. Distribute all files into training, testing and validation sets such as 80%, 10% and 10% ratio.
2. Merge the contents of the files into one file for each set, which eventually generates files such as train.txt, valid.txt, and test.txt
3. Remove extra spaces, tab spaces, and end line character. 
4. Perform tokenization of textual data in each file with custom utility, in a way that each word and punctuation is separated with just one space character.
5. Then, pass these files to GPT-2 language modeling script.

I did not use any kind of special tokens such as padding, masking, “|< endoftext >|” tokens etc in building my corpus.

Is this a right strategy? Or will there any kind of problem in this strategy?

<|||||>> Thanks for your response.
> 
> Here are the sizes of my corpus:
> 
> 1. Training set ~ 77 MB
> 2. Validation set ~ 10 MB
> 3. Testing set ~ 10 MB
> 
> I build this corpus as under:
> 
> 1. Distribute all files into training, testing and validation sets such as 80%, 10% and 10% ratio.
> 2. Merge the contents of the files into one file for each set, which eventually generates files such as train.txt, valid.txt, and test.txt
> 3. Remove extra spaces, tab spaces, and end line character.
> 4. Perform tokenization of textual data in each file with custom utility, in a way that each word and punctuation is separated with just one space character.
> 5. Then, pass these files to GPT-2 language modeling script.
> 
> I did not use any kind of special tokens such as padding, masking, “|< endoftext >|” tokens etc in building my corpus.
> 
> Is this a right strategy? Or will there any kind of problem in this strategy?

you'd better set special tokens like “<|start|>” ,'<|end|>' at the start and end of text like this "<|start|>a sentence<|end|>"。<|||||>I do not need any kind of sentence modeling for my purpose. Do I still require to specify special tokens in order to perform language modeling via GPT-2? 

Does this effect on calculating false perplexity?
<|||||>> I do not need any kind of sentence modeling for my purpose. Do I still require to specify special tokens in order to perform language modeling via GPT-2?
> 
> Does this effect on calculating false perplexity?

U can read this paper:
"Semantics of the Unwritten"
He Bai,1 Peng Shi,1 Jimmy Lin,1,2 Luchen Tan,2 Kun Xiong,2 Wen Gao,4 Jie Liu,3 Ming Li,1,2 1 David R. Cheriton School of Computer Science, University of Waterloo
2 RSVP.ai 3 Capital Normal University
4 School of Electronics Engineering and Computer Science, Peking University

And then you get it.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@mhd-git-test I have a similar problem too with GPT-2. I get a perplexity score of 7 . Did you find an answer to your problem?

