Problem with Finetuned GPT-2
Hi,
I have written this code to finetune gpt-2 on a new corpus.

```
from transformers import (
    AutoModelWithLMHead,
    AutoConfig,
    Trainer,
    AutoTokenizer,
    TextDataset,
    DataCollatorForLanguageModeling,
    TrainingArguments)

def modelTrainer(text_path, output_dir, batch_size=2, conf='gpt2', cache_dir='./Cache'):

    config = AutoConfig.from_pretrained(conf)
    model = AutoModelWithLMHead.from_config(config)
    tokenizer = AutoTokenizer.from_pretrained(conf)
    data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
    train_dataset = TextDataset(
        tokenizer=tokenizer,
        file_path=text_path,
        block_size=128,
        cache_dir=cache_dir
    )
    training_args =TrainingArguments(
        output_dir=output_dir,
        num_train_epochs=1,
        per_device_train_batch_size=batch_size,
        warmup_steps=500,
        save_steps=500,
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset,
        prediction_loss_only=True
    )

    trainer.train()
    trainer.save_model()

```

And then I use this to generate text from the finetuned model:

```
from transformers import pipeline, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('gpt2')
def textGenerator(model_dir):
    gen = pipeline('text-generation', model=model_dir, tokenizer=tokenizer)
    return gen
```

Now my problem is that even with 1 epoch a training, the quality of generated text deteriorates drastically and I get some unknown tokens in the output, like: 'Hello,�\n\n,�\n”,\n and\n,,� the\n,\n\n the,\n� Alice\n\n,“\n the\n,,,“ on”, to� she the�\n'. 
I'm guessing there is a problem with tokenizer. Can anybody help me with this?
This problem was resolved with 10 epochs of training.