Having seen quite some issue now regarding the beam scores computation, I would to clarify a bit how the scores are calculated currently for beam search and why (as noted by many of you) this is not ideal at the moment.
Let’s assume we are running the following beam search example:
import torch
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer


gpt2 = AutoModelForCausalLM.from_pretrained("gpt2", return_dict_in_generate=True)
tokenizer = AutoTokenizer.from_pretrained("gpt2")

input_ids = tokenizer("Today is a nice day", return_tensors="pt").input_ids

generated_outputs = gpt2.generate(input_ids, num_beams=2, num_return_sequences=2, output_scores=True, length_penalty=0)

Note that the length_penalty is set to 0 here which will facilitate the computation afterward since the sequence_scores are by default divided by the generated length of the sequnces since length_penalty == 1.
Now we can see that generated_outputs has the following keys:
['sequences', 'sequences_scores', 'scores']

sequences - are just the token id sequences of the 2 most probably beams.
sequence_scores - are the cumulative log probabilities of the two most probably beams. It can be formulated as a recursive formula: sequence_scores[k]_i = sequence_score[k]_{i-1} + log_probs[i-1, :])_topk(2)[k] with sequence_score[k]_{i=start_token} = 0` (i being the time step).
scores - now this is where it becomes confusing and where we should probably change the API. At the moment the scores[i, j] are defined as log_probs[i-1, j] + sequence_score[j % vocab_size]_{i-1} whereas j % vocab_size essentially defines the beam index.
Now a couple of things can be done to verify this:
generated_outputs.scores[-1].topk(2)

yields the same result as
generated_outputs.sequences_scores

Also the correct sequence id can be retrieved from the scores:
generated_outputs.scores[-1].topk(2).indices[0]

corresponds to the last generated tokens
generated_outputs.sequences[:, -1]

The problem now is that the scores don’t really give any information about the probability of token j at time i which is what most people seem to be interested in.
So, I’ll start a PR to change this behavior.