Hi,
Thanks for your interest in TrOCR! Actually, the checkpoint you are loading (i.e. microsoft/trocr-base-handwritten) is one that is already fine-tuned on the IAM dataset. So I guess further fine-tuning on this dataset is not really helpful.
Instead, it makes sense to start from a pre-trained-only checkpoint (namely microsoft/trocr-base-stage1 or microsoft/trocr-large-stage1), and fine-tune it on the IAM dataset (or another dataset of interest). I illustrate this in my notebooks here 10.