Hi @cgawron, you can take a look at my TrOCR notebooks here: Transformers-Tutorials/TrOCR at master · NielsRogge/Transformers-Tutorials · GitHub 3.
They include several example notebooks regarding fine-tuning TrOCR (which is an instance of VisionEncoderDecoderModel). I have a notebook regarding using the Seq2SeqTrainer, but also using native PyTorch. In both cases, I illustrate how to compute metrics using generate (in the notebooks I use CER, but you can easily replace it with something like ROUGE or BLEU).