High CER when Fine Tuning TrOCR Transformers to make an OCR for arabic Language
Hello Everyone ,
I need Help /Guidance regarding Creating and arabic OCR using Transformers . I'm using ViT as encoder and arabert as decoder with their pretrained weights using it like this : 
```
from transformers import VisionEncoderDecoderModel

model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(
    "google/vit-base-patch16-224-in21k", "aubmindlab/bert-base-arabertv02"
)
```
i followed this [Tutorial](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb) by [NielsRogge](https://github.com/NielsRogge) ( Thank you Niels ) . My issue is that after training i get poor results such as Caracter error rate is higher than 70% . 
ive Tried different encoders , Different decoders , Different epochs, Different learning rate , but i don't know what i miss to make it work. 
<html>
<body>
<!--StartFragment-->

Step | Training Loss | Validation Loss | Cer
-- | -- | -- | --
200 | 4.380500 | 4.674773 | 0.693091
400 | 4.213600 | 4.367142 | 0.777522
600 | 4.257300 | 4.247403 | 0.756554
800 | 3.277700 | 4.185711 | 0.712005
1000 | 2.831100 | 4.275863 | 0.748850
1200 | 2.250200 | 4.342288 | 0.788509
1400 | 2.237800 | 4.589494 | 0.768880

<!--EndFragment-->
</body>
</html>
cc @NielsRogge <|||||>Hi,

I would first try to overfit a single batch, as explained in [Karpathy's blog post](http://karpathy.github.io/2019/04/25/recipe/). This makes sure everything is set up properly.

Several folks have shown to successfully fine-tune TrOCR from pre-trained encoder + decoder checkpoints, e.g. for Japanese: https://huggingface.co/spaces/Detomo/Japanese-OCR (or this repo: https://github.com/kha-white/manga-ocr).

See also this thread for more info: https://github.com/microsoft/unilm/issues/627.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>I am having the same problem with the Bengali language as the cer is around 87%-89%. @DjouadaFarouk did your issue got solved?