gpt2-medium fine-tuned model.generate joins words and sentences together without space or newline
Hi, 

I have successfully fine-tuned and used a gpt2 model to generate text. My training corpus consist of short sentences - 3-5 words and longer ones 10-15 words. All separated by new line character. Sometimes ending with [ . ! ? ] sometimes not
  
`outputs = model.generate(
            input_ids=input_ids, max_length=max_length, temperature=temperature, 
            repetition_penalty=repetition_penalty,
            bos_token_id=tokenizer.bos_token_id,
            top_k=top_k,
            top_p=top_p
        )`

`ret = tokenizer.decode(outputs[0], skip_special_tokens=True)`

Then I fine-tuned a gpt2-medium model. The training corpus was slightly different, but structured the same as described above. 

I had to use --fp16 and --block_size=512 to fit in the GPU memory limits. 

The result:
using the fine-tuned a gpt2-medium model, I am experiencing a couple of issues:

1. I get frequent issues with lines or words stuck, without any new line or space: 
example:
**word1Word2Word3**
or:
**line 1 with some words!Another line with some words™️Next line...**

2. I get a 'warning': 
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

I've tried playing with the decode parameters with no luck:
`ret = tokenizer.decode(outputs[0], skip_special_tokens=False, clean_up_tokenization_spaces=False)`

Help appreciated, 

thanks in advance,
Albert

I can provide a google colab notebook, the fine-tuned model, the training data (or whatever needed) to show the issue. I hope it's a tokenizer issue rather than a training fault on my side, since re-training would cost a lot of cash (the training data is quite big - ~22 million lines)

 <|||||>I'm suspecting `fp16` to be the reason. Not sure whether this is supported for `generation` yet. @sshleifer - do you know more about this maybe?<|||||>A colab notebook would be great though. Or even better would be if you could upload your model to the community models :-) 
This would make it very easy for us to find the bug:
https://huggingface.co/transformers/model_sharing.html<|||||>Yeah some sort of sharing to diagnose. I don't think fp16 is the problem. What does `outputs[0]` look like? <|||||>hi, 

thanks for your reply, 

[https://huggingface.co/albertbn/gpt2-medium-finetuned-ads-fp16-blocksz512](url)

the above is the model, 

you can re-create the error using the following:

```
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained(path)
model = GPT2LMHeadModel.from_pretrained(path)

if torch.cuda.is_available():
    model.to('cuda')

input_context = '''Find a plumber nearby!'''
input_ids = torch.tensor(tokenizer.encode(input_context)).unsqueeze(0)  
if torch.cuda.is_available():
    input_ids = input_ids.cuda()

max_length=150; temperature=.175; repetition_penalty=1.3; top_k=70; top_p=0.67
outputs = model.generate(
            input_ids=input_ids, max_length=max_length, temperature=temperature, repetition_penalty=repetition_penalty,
            bos_token_id=tokenizer.bos_token_id,
            top_k=top_k,
            top_p=top_p
        )

ret = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(ret)

# Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
# Find a plumber nearby!
# Plumbing Services in Wellington NZ.
# 24/7 Emergency Plumbers Near You, Call Now For Fast Service or Repair of Your Plumbing!Need to Fix Leaking Pipes?‎NZ's #1 Gasfitter™️Call Us Today for Expert Advice & The Best Service!Get the Right Gasfitting Solution for your Home. Get It Installed Now - Free Quote Here !{KeyWord:Gas Fitting Installation}Quick And Efficient Installers

```

you can see the issue (lines stuck without \n) in the last line, starting with: 24/7 Emergency Plumbers...

thank you in advance,
Albert<|||||>> Yeah some sort of sharing to diagnose. I don't think fp16 is the problem. What does `outputs[0]` look like?

outputs[0] for the example I've posted looks like this:

```
tensor([16742,   257,   458,  4494,  6716,     0,   198,  3646, 28149,  6168,
          287, 30597, 26905,    13,   198,  1731,    14,    22, 18154,  1345,
        17024, 20173,   921,    11,  4889,  2735,  1114, 12549,  4809,   393,
        28912,   286,  3406,  1345, 28149,     0, 23037,   284, 13268,  1004,
          868,   350, 18636,    30, 48261, 37371,   338,  1303,    16, 14345,
           69,  1967,  8151, 37929, 14134,  4021,  6288,   329, 25516, 42708,
         1222,   383,  6705,  4809,     0,  3855,   262,  6498, 14345, 32232,
        28186,   329,   534,  5995,    13,  3497,   632,  2262,  4262,  2735,
          532,  3232, 19879,  3423,  5145,    90,  9218, 26449,    25, 39699,
          376,  2535, 32588,    92, 21063,   843,   412,  5632, 15545,   364],
       device='cuda:0')
```
there is no white space separating 0, 23037 (23037 is the only index in the output: Plumbing**!Need** )
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Regarding the second question

>     2. I get a 'warning':
>        Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

It's explained [here](https://jaketae.github.io/study/gpt2/#setup): "For open-end generation, HuggingFace will set the padding token ID to be equal to the end-of-sentence token ID". Code is here: https://github.com/huggingface/transformers/blob/b880508440f43f80e35a78ccd2a32f3bde91cb23/src/transformers/generation_utils.py#L410-L414