Hello,
I am using beam search with a pre-trained T5 model for summarization. I would like to visualize the beam search process by showing the tokens with the highest scores, and eventually the chosen beam like this diagram:
image894×669 56.8 KB
(Taken from How to generate text: using different decoding methods for language generation with Transformers 5)
I am unsure how I can show the tokens and their corresponding scores.
I followed the discussion [Announcement] GenerationOutputs: Scores, Attentions and Hidden States now available as outputs to generate 6 and https://github.com/huggingface/transformers/pull/9150.
Following the docs, upon calling generate, I have set return_dict_in_generate=True, output_scores=True
generated_outputs = model_t5summary.generate(
  input_ids=input_ids.to(device),
  attention_mask=features['attention_mask'].to(device),
  max_length=input_ids.shape[-1] + 2,
  return_dict_in_generate=True,
  output_scores=True,
  output_hidden_states=True,
  output_attentions=True,
  no_repeat_ngram_size=2,
  early_stopping=True,
  num_return_sequences=3,
  num_beams=5,
)

Now I have an instance of BeamSearchEncoderDecoderOutput.
If I understand the docs (Utilities for Generation — transformers 4.2.0 documentation 5) correctly, scores will provide me with what I want but I am unsure on how to use the scores.
Any help/pointers from the community would be greatly appreciated, thank you