This PR: https://github.com/huggingface/transformers/pull/9150 19 added GenerationOutputs similar to ModelOutpus for Pytorch Transformers.
=> Check out the corresponding tweet with an example: https://twitter.com/SimonBrandeis/status/1346858472000937984 55 .
The community has been asking for quite some time for this feature, see

https://github.com/huggingface/transformers/issues/7654 17
https://github.com/huggingface/transformers/issues/8656 7
https://github.com/huggingface/transformers/issues/3891 7

When setting return_dict_in_generate the PyTorch .generate() method now returns GenerationOutputs that are very similar in style to ModelOutputs. The usual generation output_ids can then be accessed under outputs["sequences"].
For all “non-beam search” generations (greedy_search and sample), one has now access to

all attentions of every layer at every generation step (be careful memory might blow up here) if output_attentios=True

all hidden_states of every layer at every generation step if output_hidden_states=True.

scores. Now the scores correspond to the processed logits -> which means the models lm head output after applying all processing functions (like top_p or top_k or repetition_penalty) at every generation step in addition if output_scores=True.

For all “beam_search” methods:

all attentions and all hidden_states of every layer at every generation step if output_attentions and output_hidden_states are set to True


scores now correspond to all processed lm head logits + the current beam_scores for each output token. This score (next_token_scores + beam_scores) is the most important score at every generation step so we decided to output this score.

sequence_scores In addition to the three outputs above for beam search output_scores=True also returns the final “beam score” for each returned sequence (see https://twitter.com/SimonBrandeis/status/1346858472000937984 55)

This should make it easier to analyze the generation of transformer models and should also allow the user to build “confidence” graphs from the scores and sequence_scores.
For more in-detail information please check out the docs: https://huggingface.co/transformers/master/internal/generation_utils.html#generate-outputs 51
We would be very happy for some feedback from the community if the GenerationOutputs meets the expectations .