We found the problem: the shape of the attention matrix was not correct.
Here are the modifications
# From T5 documentation
# Initial shape: Tuple (one element for each generated token) of tuples (one element for
# each layer of the decoder) of torch.FloatTensor of shape
# (batch_size, num_heads, generated_length, sequence_length).

# combine all cross attention into one tensor
x = [torch.stack(beam_outputs['cross_attentions'][i]) for i in range(len(beam_outputs['cross_attentions']))]
x = torch.stack(x)
# Shape: (nb_generated, nb_layer, (batch_size, num_heads, generated_length, sequence_length))
print(x.shape)

x = x.transpose(1,0)
# (nb_layer, nb_generated, batch_size, num_heads, generated_length, sequence_length)
print(x.shape)

x = x.transpose(1,3)
# (nb_layer, num_heads, batch_size, nb_generated, generated_length, sequence_length)
print(x.shape)

x = torch.squeeze(x, 4)
# (nb_layer, num_heads, batch_size, nb_generated, sequence_length)
print(x.shape)

x = x.transpose(2, 1)
# (nb_layer, batch_size, num_heads, nb_generated, sequence_length)
print(x.shape)
cross_attentions = x

We can compute the encoder/decoder tokens:
encoder_text = tokenizer.convert_ids_to_tokens(input_ids[0])
decoder_text = tokenizer.convert_ids_to_tokens(beam_outputs.sequences[0])

encoder_tokens=np.array(encoder_text)
decoder_tokens=np.array(decoder_text[:-1])

The initial sentence is: In Belgium, summers can be very dry and the heat is burning. and the paraphrase: 'In Belgium the summers can be very dry and the heat is burning.
The associated tokens are:
['▁para', 'phrase', ':', '▁In', '▁Belgium', ',', '▁summer', 's', '▁can', '▁be', '▁very', '▁dry', '▁and', '▁the', '▁heat', '▁is', '▁burning', '.', '</s>']
['<pad>', '▁In', '▁Belgium', '▁the', '▁summer', 's', '▁can', '▁be', '▁very', '▁dry', '▁and', '▁the', '▁heat', '▁is', '▁burning', '.', '</s>']

Here are the attention score sorted by importance:
layer = 0
head = 0 # choose head to analyze
att = cross_attentions[layer, 0, head]
att.shape

for i in range(att.shape[0]):
    idx = np.argsort(att[i].cpu().numpy())[::-1][:6]
    print(f"Predicted token: {decoder_tokens[i]}, input related tokens: {encoder_tokens[idx]}")

Here are some results:
Predicted token: ▁In, input related tokens: ['</s>' '▁In' 'phrase' '.' ':' '▁Belgium']
Predicted token: ▁Belgium, input related tokens: ['▁Belgium' '</s>' 'phrase' ':' '▁para' '▁summer']
Predicted token: ▁the, input related tokens: ['</s>' 'phrase' ',' '.' '▁the' ':']
Predicted token: ▁summer, input related tokens: ['▁Belgium' '</s>' 'phrase' '▁In' 's' '▁summer']

We solved this problem with the help of the decoder encoder version of bertviz 5