Generation with gpt-2
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Hi @julien-c and all,
we are developing a training with GPT-2 language model based on our custom dataset.
Firstly, we train a custom Tokenizer with tokenizers ( as described [here](https://github.com/huggingface/tokenizers/issues/166) ) .
Now I have a dummy GPT-2 model trained on 1 epoch but my problem is on inference.
The result is always the same when I feed the model with the starting text.

Code (is the same at link ) 
```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("my-gpt2-tokenizer")
model = GPT2LMHeadModel.from_pretrained('my-gpt2-model')

generated = tokenizer.encode("The Manhattan bridge")
context = torch.tensor([generated])
past = None

for i in range(100):
    print(i)
    output, past = model(context, past=past)
    token = torch.argmax(output[..., -1, :])

    generated += [token.tolist()]
    context = token.unsqueeze(0)

sequence = tokenizer.decode(generated)
print(sequence)
```
Using this code, the output is repeated:

```
The Manhattan bridge<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit
```
I think that problems are two basically:
1. the first is that for every inference we take only the argument that maximizes probability, and I don't use any type of sampler based on some type of distribution.
2. the second is that the content fed after the first step is always the next word predicted , so it goes in loop.

I don't have any other ideas. Is there an example of inference with huggingface lib?
Thanks


There is an example: https://github.com/huggingface/transformers/blob/master/examples/run_generation.py<|||||>Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
