Help with TrOCR training for Spanish
### System Info

I'm using Torch 2.0 and Transformers 4.28.0 running in a Google Colab




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi 

Thanks @NielsRogge for your incredible work in Transformers. 

I'm working in develop a handwritten system recognition  in spanish, so i choose TrOCR for the traduction from handwrite to text in spanish.

I think I followed yours Notebooks examples to fine tune and inference with TrOCR and many post with people with the same problem when we need to train TrOCR in a diferent language, spanish for me.



The code to create dataset is:

 
class SpanishDataset(Dataset):
    

     def __init__(self, root_dir, df, processor, max_target_length=128):
        self.root_dir = root_dir
        self.df = df
        self.processor = processor
        self.max_target_length = max_target_length

     def __len__(self):
         return len(self.df)

     def __getitem__(self, idx):
         # get file name + text 
        file_name = self.df['Path'][idx]
        text = self.df['Text'][idx]
        # prepare image (i.e. resize + normalize)
        image = Image.open(self.root_dir + file_name).convert("RGB")
        pixel_values = self.processor(image, return_tensors="pt").pixel_values
        # add labels (input_ids) by encoding the text
        labels = self.processor.tokenizer(text, padding="max_length",max_length=self.max_target_length).input_ids
        # important: make sure that PAD tokens are ignored by the loss function
        labels = [label if label != self.processor.tokenizer.pad_token_id else -100 for label in labels]

        encoding = {"pixel_values": pixel_values.squeeze(), "labels": torch.tensor(labels)}
        return encoding


and 
 
```
 feature_extractor = AutoFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
decoder_tokenizer = AutoTokenizer.from_pretrained("bertin-project/bertin-roberta-base-spanish")

processor =TrOCRProcessor(feature_extractor=feature_extractor, tokenizer=decoder_tokenizer)

processor.save_pretrained('./processor')

processor = TrOCRProcessor.from_pretrained("./processor")


train_dataset = SpanishDataset(root_dir='/ShardDrives/MyDrive',
                           df=train_df,
                           processor=processor)
eval_dataset = SpanishDataset(root_dir='/ShardDrives/MyDrive',
                           df=test_df,
                           processor=processor) 
```

I  use this encoder and decoder:


```
model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(
    "google/vit-base-patch16-224-in21k", "bertin-project/bertin-roberta-base-spanish")

 
# set decoder config to causal lm
model.config.decoder.is_decoder = True
model.config.decoder.add_cross_attention = True
# set special tokens used for creating the decoder_input_ids from the labels
model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id
# make sure vocab size is set correctly
model.config.vocab_size = model.config.decoder.vocab_size

# set beam search parameters
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.max_length = 64
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 4


# ensure that randomly initialized cross-attention layers are added
assert model.config.decoder.is_decoder is True
assert model.config.decoder.add_cross_attention is True
```

I'm using cer as metric 

```
def compute_metrics(pred):
    labels_ids = pred.label_ids
    pred_ids = pred.predictions

    pred_str = processor.batch_decode(pred_ids, skip_special_tokens=True)
    labels_ids[labels_ids == -100] = processor.tokenizer.pad_token_id
    label_str = processor.batch_decode(labels_ids, skip_special_tokens=True)

    cer = cer_metric.compute(predictions=pred_str, references=label_str)

    return {"cer": cer}
```

and the training code is:

```
training_args = Seq2SeqTrainingArguments(
 
    evaluation_strategy="epoch",
    learning_rate=2e-4,    
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=100,
    output_dir="./",
    predict_with_generate=True,  
)   

# Training
trainer = Seq2SeqTrainer(
    model=model,
    tokenizer=processor.feature_extractor,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=default_data_collator
)
trainer.train()
```

But the Cer is bad (> 0.9) 

This is an output for a dataset element

```

train_dataset[0]

{'pixel_values': tensor([[[ 0.9922,  1.0000,  1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          ...,
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000]],
 
         [[ 0.9922,  1.0000,  1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          ...,
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000]],
 
         [[ 0.9922,  1.0000,  1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          [ 0.9843,  0.9922,  0.9922,  ..., -1.0000, -1.0000, -1.0000],
          ...,
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
          [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000]]]),
 'labels': tensor([    0,  1323,   344,  2858, 11966,    66, 11507,  3298,   344, 14783,
            66,     2,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,
          -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100])}
```

I have an error, but i don't know where, so any help or advice  is welcome.

Thank very much for everything

### Expected behavior

I expect to train a handwrite recognition system for spanish using TrOCR
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.