Solved by nielsr in post #2 
              
              
                Hi, 
Thanks for your interest in TrOCR! Actually, the checkpoint you are loading (i.e. microsoft/trocr-base-handwritten) is one that is already fine-tuned on the IAM dataset. So I guess further fine-tuning on this dataset is not really helpful. 
Instead, it makes sense to start from a pre-trained-on…