How to avoid adding  double start of token<s><s> in TrOCR during training ?
**Describe the bug**
The model I am using (TrOCR Model):

The problem arises when using:
* [x] the official example scripts: done by the nice tutorial  [(fine_tune)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb) @NielsRogge 
* [x] my own modified scripts: (as the script below )
```


processor = TrOCRProcessor.from_pretrained("microsoft/trocr-large-handwritten")
def compute_metrics(pred):
    labels_ids = pred.label_ids
    print('labels_ids',len(labels_ids), type(labels_ids),labels_ids)
    pred_ids = pred.predictions
    print('pred_ids',len(pred_ids), type(pred_ids),pred_ids)
    pred_str = processor.batch_decode(pred_ids, skip_special_tokens=True)
    print(pred_str)
    labels_ids[labels_ids == -100] = processor.tokenizer.pad_token_id
    label_str = processor.batch_decode(labels_ids, skip_special_tokens=True)
    print(label_str)
    cer = cer_metric.compute(predictions=pred_str, references=label_str) 
    return {"cer": cer}
 

class Dataset(Dataset):
    def __init__(self, root_dir, df, processor, max_target_length=128):
        self.root_dir = root_dir
        self.df = df
        self.processor = processor
        self.max_target_length = max_target_length

    def __len__(self):
        return len(self.df)

    def __getitem__(self, idx):
        # get file name + text 
        file_name = self.df['file_name'][idx]
        text = self.df['text'][idx]
        # prepare image (i.e. resize + normalize)
        image = Image.open(self.root_dir + file_name).convert("RGB")
        pixel_values = self.processor(image, return_tensors="pt").pixel_values
        # add labels (input_ids) by encoding the text
        labels = self.processor.tokenizer(text, 
                                          padding="max_length",
					                      truncation=True, 
                                          max_length=self.max_target_length).input_ids
        # important: make sure that PAD tokens are ignored by the loss function
        labels = [label if label != self.processor.tokenizer.pad_token_id else -100 for label in labels]
        # encoding  
        return {"pixel_values": pixel_values.squeeze(), "labels": torch.tensor(labels)}


         #Train a model
        model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-large-handwritten")
        # set special tokens used for creating the decoder_input_ids from the labels
        model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
        model.config.pad_token_id = processor.tokenizer.pad_token_id
        # make sure vocab size is set correctly
        model.config.vocab_size = model.config.decoder.vocab_size
        # set beam search parameters
        model.config.eos_token_id = processor.tokenizer.sep_token_id
        model.config.max_length = 64 
        model.config.early_stopping = True
        model.config.no_repeat_ngram_size = 3
        model.config.length_penalty = 2.0
        model.config.num_beams = 4

        model.config.decoder.is_decoder = True 
        model.config.decoder.add_cross_attention = True
        working_dir =  './test/'
        training_args = Seq2SeqTrainingArguments(...)
        # instantiate trainer
        trainer = Seq2SeqTrainer(model=model,  args=training_args, train_dataset = train_dataset,
                 data_collator = default_data_collator, )
    trainer.train()
# python3 train.py path/to/labels  path/to/images/
```
- Platform: Linux  Ubuntu  distribution  [GCC 9.4.0] on Linux 
- PyTorch version (GPU?):  0.8.2+cu110 
- transformers: 4.22.2 
- Python version:3.8.10

A clear and concise description of what the bug is.
To **Reproduce** Steps to reproduce the behavior: 
1. After training the model or during the training phase  when evaluating metrics calculate I see the model added the double start of token `<s><s>` or ids `[0,0, ......,2,1,1, 1 ]`
2. here is an example during the training phase showing generated tokens in compute_metrics 
Input predictions: `[[0,0,506,4422,8046,2,1,1,1,1,1]]` 
Input references: `[[0,597,2747 ...,1,1,1]] `
4. Other examples during testing models [[![enter image description here](https://i.stack.imgur.com/sWzbf.png)](https://i.stack.imgur.com/sWzbf.png)]

**Expected behavior** A clear and concise description of what you expected to happen. 
In 2 reproduced problems: 
I am expecting during `training`Input predictions should be: `[[0,506,4422, ... ,8046,2,1,1,1,1,1 ]] `

In addition during the `testing` phase:  generated text without double` **<s><s>** `
`tensor([[0,11867,405,22379,1277,..........,368,2]]) `
`<s>ennyit erről, tőlem fényképezz amennyit akarsz, a véleményem akkor</s>`
cc @ArthurZucker
cc @ArthurZucker <|||||>+1<|||||>Thanks for Natabara for his comment The solution is super easy by just skipping <s> start token `labels= labels[1:] ` coming from the tokenizer because the tokenizer adds start token <s> and the TrOCR adds start token <s> automatically as mentioned in TrOCR paper ``` def __getitem__(self, idx): # get file name + text file_name = self.df['file_name'][idx] text = self.df['text'][idx] # prepare image (i.e. resize + normalize) image = Image.open(self.root_dir + file_name).convert("RGB") pixel_values = self.processor(image, return_tensors="pt").pixel_values # add labels (input_ids) by encoding the text labels = self.processor.tokenizer(text, padding="max_length", truncation=True, max_length=self.max_target_length).input_ids # important: make sure that PAD tokens are ignored by the loss function labels = [label if label != self.processor.tokenizer.pad_token_id else -100 for label in labels] labels= labels[1:] return {"pixel_values": pixel_values.squeeze(), "labels": torch.tensor(labels)} ```