[Bug/Question] Write With Transformers Implementation vs. Custom Implementation
Hi, 

Not sure if this is a bug or perhaps a misimplementation, but i'm comparing the results of using gpt2-large on the 'Write With Transformers' Text Generation example - https://transformer.huggingface.co/doc/gpt2-large, vs. my own implementation of the text generation tool. My use-case is to generate distractor options in an MCQ environment, given a few (15-20) prior examples for the style of distractor questions to generate. 

The general format I am implementing is:
`Question: _______ . Answer 1: ___<the correct answer>___ . Answer 2: _____<distractor 1>_____ . Answer 3: ____<distractor 2>_____`

## Write With Transformers
full doc available here - https://transformer.huggingface.co/share/CZqVXdngic

### Model Config
Model size - gpt2/large
Top-p - 0.9
Temperature - 1
Max time - 2.3 
### Output
On the 'Write With Transformers' page, I write in the examples: 
eg. `Question: How is calorie related to the S.I unit of that quantity?Answer 1: 1 cal = 4.2 J.Answer 2: 1 cal = 3.2 J.Answer 3: 1 cal = 10 J.`

and when I try to generate predictions for the following Question-Answer pair: 
`Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W.`

am able to generate a few solid distractors: 
`Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2:  1 H. P. = 1. 3 W.  Answer 3 : 1 H . P. =  1. 5 W.`

## Custom Case

### Output
Here's how I create a dataset from the set of questions I've initially written myself. 
```
def make_dataset(dataset, epochs):
    total_text = '<|endoftext|>'
    qa = [t for t in dataset]
    for _ in range(epochs):
        random.shuffle(qa)
        total_text += '<|endoftext|>'.join(qa) + '<|endoftext|>'
    return total_text
```

This is the training model params:

```
!python run_language_modeling.py \
    --output_dir=output/$handle \
    --overwrite_output_dir \
    --overwrite_cache \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train --train_data_file=$handle\_train.txt \
    --logging_steps 20 \
    --per_gpu_train_batch_size 1 \
    --num_train_epochs 15
```

num_return_sequences = 5

```
for start in SENTENCES:
    val = !python run_generation.py \
        --model_type gpt2 \
        --model_name_or_path output/$handle \
        --length 40 \
        --num_return_sequences $num_return_sequences \
        --temperature 0.23 \
        --p 0.95 \
        --seed $seed \
        --prompt {'"<|endoftext|>' + start + '"'}
    generated = [val[-1-2*k] for k in range(num_return_sequences)[::-1]]
    print(f'\nStart of sentence: {start}')
    for i, g in enumerate(generated):
        g = g.replace('<|endoftext|>', '')
        print(f'* Generated #{i+1}: {g}')
```

These are my generated mcq pairs:

**Generated #1: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**
**Generated #2: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**
**Generated #3: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: ______________ = 905 J.Answer 3: 50 J.**
**Generated #4: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**
**Generated #5: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
