Training a GPT-2 from scratch in Greek-text, results in a low perplexity score of 7 after 15 epochs. Is it normal that score?
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

I try to train a GPT-2 from scratch in Greek with an older version of run_language_modeling.py (https://github.com/huggingface/transformers/tree/master/examples/language-modeling) script, but I get a low perplexity score of 7 after 15 epochs.

My data for train is about 4.6Gb and is constructed as 5 sentences per line. The data for the evaluation is about 450Mb constructed with the same way. Use of BPE for the encoding with a vocab of 22000 merges.
For the train is used FP16 O2.
<|||||>I think it is normal.
For perplexity, the lower, the better.
By the way, can you share your script for preprocessing data?<|||||>@xx-zhou16 Ok i found a mistake i had...when i was computing the loss i wasn't ignoring the pad_token . Now i train it again , i think now the perplexity score will stop about 11, which is again low. 
I use the line by LineByLineTextDataset :

class LineByLineTextDataset(Dataset):
    def __init__(self, tokenizer: PreTrainedTokenizer, args, file_path: str, block_size=512):
        assert os.path.isfile(file_path)
        logger.info("Creating features from dataset file at %s", file_path)

        with open(file_path, encoding="utf-8") as f:
            lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
        self.examples = tokenizer.batch_encode_plus(lines, add_special_tokens=True, truncation=True, max_length=block_size)["input_ids"]    
 
    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):
        return torch.tensor(self.examples[i], dtype=torch.long)

    def load_and_cache_examples(args, tokenizer, evaluate=False):
        file_path = args.eval_data_file if evaluate else args.train_data_file
        if args.line_by_line:
            return LineByLineTextDataset(tokenizer, args, file_path=file_path, block_size=args.block_size)

padding and creating an attention mask .

def collate(examples: List[torch.Tensor]):
        padding_value = 0 if tokenizer._pad_token is None else tokenizer.pad_token_id
        input_ids = pad_sequence(examples, batch_first=True, padding_value=padding_value)
        max_length = input_ids.shape[1]
        attention_mask = torch.stack([torch.cat([torch.ones(len(t), dtype=torch.long), torch.zeros(max_length - len(t), dtype=torch.long)]) for t in examples])
        return input_ids, attention_mask


and that is in the evaluation , for the perplexity score computation.

    for batch in tqdm(eval_dataloader, desc="Evaluating"):
        input_ids, attention_mask = batch
        inputs, labels = (input_ids, input_ids)

        inputs = inputs.to(args.device)
        labels = labels.to(args.device)
        attention_mask = attention_mask.to(args.device)

        with torch.no_grad():
            outputs = model(inputs, labels=labels, attention_mask=attention_mask)
            lm_loss = outputs[0].mean().item()
            eval_loss += lm_loss
        nb_eval_steps += 1


    eval_loss = eval_loss / nb_eval_steps
    perplexity = torch.exp(torch.tensor(eval_loss))<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
