Unexpected sequences_scores in BeamSearchDecoderOnlyOutput
## Environment info

- `transformers` version: 4.11.3
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.0+cu111 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@qqaatw 

## Information

This is a follow up to issue https://github.com/huggingface/transformers/issues/14065. I calculate the probability of each generated token conditional on the previous generated tokens as suggested in that previous issue:

```python
import torch

from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("gpt2", return_dict_in_generate=True)
tokenizer = AutoTokenizer.from_pretrained("gpt2")

input_ids = tokenizer("Today is a nice day", return_tensors="pt").input_ids

generated_outputs = model.generate(input_ids, num_beams=2, max_length=8, output_scores=True)
gen_ids = generated_outputs["sequences"][0, input_ids.shape[-1]:]
vocab_size = generated_outputs["scores"][0].shape[-1]


print(gen_ids) # tensor([ 11, 475, 314])

# Here we find out at each time-step, which beam each generated id belongs to.
values, indices = torch.topk(generated_outputs["scores"][0].view(-1), k=2)
print(values, (indices % vocab_size), (indices / vocab_size).long()) # tensor([-1.5148, -1.8792]) tensor([329,  11]) tensor([0, 0])
values, indices = torch.topk(generated_outputs["scores"][1].view(-1), k=2)
print(values, (indices % vocab_size), (indices / vocab_size).long()) # tensor([-3.6481, -3.9508]) tensor([475, 262]) tensor([1, 0])
values, indices = torch.topk(generated_outputs["scores"][2].view(-1), k=2)
print(values, (indices % vocab_size), (indices / vocab_size).long()) # tensor([-5.6957, -5.8748]) tensor([314, 340]) tensor([0, 0])

# So we know at time-step 1, the token id `475` belongs to beam 1, others belong to beam 0.

logprob_gen_token0 = generated_outputs["scores"][0][0, gen_ids[0]]
logprob_gen_token1 = generated_outputs["scores"][1][1, gen_ids[1]] - logprob_gen_token0
logprob_gen_token2 = generated_outputs["scores"][2][0, gen_ids[2]] - logprob_gen_token0 - logprob_gen_token1

print(logprob_gen_token0.exp(), logprob_gen_token1.exp(), logprob_gen_token2.exp()) 

# Outputs:
# tensor(0.1527) tensor(0.1705) tensor(0.1290)
```

Now I look at the `sequences_scores`:

```python
# tensor([0.4907])
generated_outputs["sequences_scores"].exp()
```

I would have expected the `sequences_scores` to be equal to the product of the probabilities of each generated token conditional on the previous generated tokens: 0.1527 * 0.1705 * 0.1290 = 0.0034.

I'm probably misunderstanding something. Thanks in advance for your help!

## To reproduce

See this colab:
https://colab.research.google.com/drive/11rRAFuNycLLDiDDwU02mBgXjpBOXCe4P#scrollTo=55CjTHwLc_gE

## Expected behavior

I would have expected the `sequences_scores` to be equal to the product of the probabilities of each generated token conditional on the previous generated tokens: 0.1527 * 0.1705 * 0.1290 = 0.0034
```
# Get average of log probs.
sum_logprob = sum((logprob_beam0_gen_token0, logprob_beam0_gen_token1, logprob_beam0_gen_token2))
seq_scores = (sum_logprob / generated_outputs["sequences"].shape[-1])

print(seq_scores == generated_outputs["sequences_scores"][0]) # True
```<|||||>Thanks for the clarification! Do you know of any references that explain why the sequences_scores is defined this way? If we compare generations for different prompts using this definition, then a generation with a longer prompt may get a lower score even if it is more confident in its generated tokens than a generation with a shorter prompt and less confidence in its generated tokens. Obviously, I can calculate different scores for myself, but just wondering about the motivation for this definition. Thanks again!<|||||>Good question! 

Based on my understanding, the conditional generation functionality of `transformers` can be used on pure decoder architectures like `GPT2` or encoder-decoder architectures such as `Bert2Bert`. 

In the former case, the prompt is concatenated with generated outputs as that is how CLM works, so you might be wondering if the prompt is longer, the sequence score will be lower. However, that is based on an assumption that different pairs of prompt and generated outputs are being compared, and I'm not 100% sure whether this direct comparison is hypothetically reasonable. 

In the later case, a prompt is separated from generated outputs, i.e. the prompt is firstly inputted into an encoder, and then via cross-attention, a decoder amalgamates encoder outputs and decoder inputs to generate output_ids autoregressively. Therefore, because the prompt does not exist in decoder outputs, this calculation works as you expected, i.e. the sequence score is not affected by the prompt's length.

Maybe this calculation can be adjusted to fit your use-case. That is, if the prompt is concatenated with generated outputs, then we only use generated sequence length to calculate averaged log probability.

cc @patrickvonplaten <|||||>Thanks for answering so far @qqaatw!

A couple of things here @hacobe,

it seems like you are using `beam_sample` with `gpt2` (`do_sample=True` + `num_beams > 1`) which is a very exotic use case. Since you are essentially sampling the output token each time, you can't assume that the `topk()` will return you the actual sampled token. 

Instead you should rather use `generate(...)` with `do_sample=False` and check again<|||||>Thanks for taking a look at this @patrickvonplaten!

I don't set `do_sample` in `generate(...)` and it defaults to `False`. Does your comment still apply?<|||||>By the way, my goal is just to get the probabilities of the tokens that are generated from GPT2 using beam search. The code snippet in the "Information" section of the first comment was suggested by @qqaatw in a [previous issue](https://github.com/huggingface/transformers/issues/14065).<|||||>See: https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175/15?u=patrickvonplaten

I'll improve the behavior for beam search<|||||>https://github.com/huggingface/transformers/pull/14654<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.