greedy beam search generates same sequence N times
## ❓ Questions & Help

I am doing greedy beam search (without sampling to avoid randomness) using GPT-2. However, all the returned sequences are same. Why is that the case? Shouldn't it give N best and different sequences?

```python
model = GPT2LMHeadModel.from_pretrained('gpt2-medium').cuda()
tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')

input_context = 'The dog'
input_ids = torch.tensor(tokenizer.encode(input_context)).unsqueeze(0).cuda()
outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3)
for i in range(3):
    print('Generated {}: {}'.format(i, tokenizer.decode(outputs[0][i], skip_special_tokens=True)))
```

The resulting output is:
```
Generated 0: The dog was taken to a veterinary clinic for treatment.

The dog's owner said the!
Generated 1: The dog was taken to a veterinary clinic for treatment.

The dog's owner said the!
Generated 2: The dog was taken to a veterinary clinic for treatment.

The dog's owner said the!
````
Took me most of the day to figure this out, set the `do_sample` arg to `True`

```
outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3, do_sample=True)
```<|||||>Actually, I don't want to do sampling because that is random and would give different results each time I run for the same prompt.

I am looking for greedy beam search which should be able to give the static top sequences (which is not happening)! Sad.<|||||>In your case your have 3 parallel beam search going on with a beam of 5 in each case.

But the current beam search only returns the top beam in each case, we don't have an option to return all beams.<|||||>Thanks Thomas. Does that mean `num_return_sequences` is only useful when `do_sample` is `True`?<|||||>That's a good point.

We could probably take the `num_return_sequences` top beams in the case of having beam search + greedy decoding otherwise this option is not useful in this case.
<|||||>Thanks @thomwolf for the clarification. So, in case of greedy decoding, you would do beam search only once and take the top ```num_return_sequences``` ones. 

Any rough ETA you have?
<|||||>No ETA, but if you need it now, feel free to make a PR and I or Lysandre will give a look<|||||>Hi, I'm also interested in this feature - did anyone attempt to implement greedy beam search that returns multiple sequences? (Alternatively, has a an idea of which part of the function should be fixed, so I can try it myself)? Thanks!<|||||>See PR #3078 for how this feature is implemented.

The following example: 
```
model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

input_context = 'The dog'
input_ids = torch.tensor(tokenizer.encode(input_context)).unsqueeze(0)
outputs = model.generate(input_ids=input_ids, num_beams=20, num_return_sequences=3, do_sample=False)
for i in range(3):
    print('Generated {}: {}'.format(i, tokenizer.decode(outputs[i], skip_special_tokens=True)))
```

would produce:

```
Generated 0: The dog was taken to a local hospital, where he was pronounced dead.

The dog was
Generated 1: The dog was taken to a local hospital, where it was treated and released.

The dog
Generated 2: The dog was taken to a local hospital where he was pronounced dead.

The dog's owner
```

<|||||>Thanks @patrickvonplaten for your effort in this. 