return_dict scores are inconsistent between sampling and beam search
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-5.4.0-1035-aws-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@SBrandeis @patrickvonplaten 

## Information

When generating text using `model.generate` and `return_dict_in_generate` and beam search, beam search [sets the score for the first token to be 1e-9 across all beams other than the first](https://github.com/huggingface/transformers/blob/a449ffcbd2887b936e6b70a89e533a0bb713743a/src/transformers/generation_utils.py#L1576). This is not consistent with the sampling score which maintains the same score for the same token across different generations, and results in rather confusing behavior for score when using beam search.

## To reproduce

Steps to reproduce the behavior:

```python
import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
text = "How are"
input_ids = tokenizer.encode(text, return_tensors='pt')

# Beam Search
generated_outputs = model.generate(input_ids, return_dict_in_generate=True, output_scores=True, num_return_sequences=4, num_beams=4, max_length=input_ids.shape[-1] + 2)

gen_sequences = generated_outputs.sequences[:, input_ids.shape[-1]:]
# tensor([[ 345, 1804],
#        [ 345, 1016],
#        [ 356, 1016],
#        [ 345, 4203]])
probs = torch.stack(generated_outputs.scores, dim=1).softmax(-1)
gen_probs = torch.gather(probs, 2, gen_sequences[:, :, None]).squeeze(-1)
# tensor([[3.7034e-01, 1.4759e-01],
#        [1.9898e-05, 2.7981e-01],
#        [1.9898e-05, 3.2767e-05],
#        [1.9898e-05, 2.9494e-03]])

np.random.seed(42)
torch.manual_seed(42)

# Sampling
generated_outputs = model.generate(input_ids, return_dict_in_generate=True, output_scores=True, num_return_sequences=3, do_sample=True, top_p=.9, max_length=input_ids.shape[-1] + 2)

gen_sequences = generated_outputs.sequences[:, input_ids.shape[-1]:]
probs = torch.stack(generated_outputs.scores, dim=1).softmax(-1)
gen_probs = torch.gather(probs, 2, gen_sequences[:, :, None]).squeeze(-1)

gen_sequences
# tensor([[ 262, 1180],
#         [ 345, 1016],
#         [ 345, 4203]])

gen_probs[:, 0]
# tensor([0.1121, 0.5147, 0.5147])
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

When performing beam search the score from the model's logits should be output as the score rather than 1e-9.

Hey @mshuffett, 

I understand your concern and I think I agree with you! Would you be interested in opening a PR to fix this for `beam_search` and `group_beam_search` ? It's essentially just moving the ` + beam_scores` line further down<|||||>@patrickvonplaten I would be happy to but upon trying your suggested fix, it does solve the problem with the score of the first token, but the second token now has the same problem.

I just moved this line below the `return_dict_in_generate` block.
```python
next_token_scores = next_token_scores + beam_scores[:, None].expand_as(next_token_scores)
```

I believe this is still unexpected but I'm not yet sure why this is happening. If you have any thoughts that would be helpful.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Unstale<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Ping
<|||||>Yeah I'm probably not going to be able to submit a PR for this, but I do
think it should be fixed.
ᐧ

On Mon, May 31, 2021 at 2:27 AM Patrick von Platen ***@***.***>
wrote:

> Ping
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/10012#issuecomment-851265135>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAIFLUJ6IGHT52SRTDKJIKDTQM25NANCNFSM4XDNCCEQ>
> .
>


-- 
Michael Shuffett
Written with compose.ai
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>I had the same issue. I'm trying to replicate OpenAI's behavior when selecting "Show probabilities: Full Spectrum". I would like, at each step, for it to show the top N candidate tokens as well as all their probabilities. 

I've not quite figured out how to make it select the _top_ candidates, because sampling is selecting them randomly. It seems like beam search does indeed do what I want selecting always the top probabilities, but then I can't get the true probabilities this way because of the 1e-09 issue. <|||||>@monsieurpooh,

Could you open a new issue for this one? I'm not quite sure whether you are interested in sampling, beam search, etc... :-) Happy to extend `generate()` to cover more important use cases<|||||>The only thing I was trying to do was get the top 10 tokens (and their probabilities) for the next 1 token. For example: "I went to the" -> {" store": 0.25, " park": 0.1, ...} 

I do not need any beam search, sampling, etc. I used the workaround described earlier in the thread and it works perfectly.<|||||>@patrickvonplaten can you please this question [here](https://stackoverflow.com/questions/72180737/beam-search-and-generate-are-not-consistent) . If I don't miss something, I think there is a bug in beam_search. Thanks<|||||>Hey @rafikg,

Could you please open a new issue or use the forum: https://discuss.huggingface.co/ if you have a question? Thanks!<|||||>@monsieurpooh can you please elaborate how you implemented the workaround?

Looking at master during the time of your posting in posting in March 7th, the workaround you mentioned seems already implemented?

https://github.com/huggingface/transformers/blob/5c6f57ee75665499c8045a8bf7c73bf2415fba20/src/transformers/generation_utils.py#L2112

`next_token_scores` is not invoked until after the `if return_dict_in_generate` https://github.com/huggingface/transformers/blob/5c6f57ee75665499c8045a8bf7c73bf2415fba20/src/transformers/generation_utils.py#L2115

`next_token_scores_processed` is being used instead