To calculate generative metrics during training either clone Patrics branch or Seq2SeqTrainer PR branch.
The default Trainer returns the output of the final LM head layer which is why the shape is  batch_size * max_output_len * vocab_size. The above branches instead calls the generate method inside the eval loop and returns the generated ids as EvalPrediction.predictions and the actual labels as EvalPrediction.label_ids, which then you can pass to the tokenizers decode or batch_decode method in the compute_metrics function  to get the text and calculate metrics like ROUGE, BLUE etc