M2M100 training does not improve model's performance
### System Info

- `transformers` version: 4.23.0.dev0
- Platform: Linux-5.4.0-1072-aws-x86_64-with-glibc2.27
- Python version: 3.8.13
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@patil-suraj

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hello,
I’m trying to fine-tune M2M100 using the  example script run_translation.py and it seems that the model is not improving.
I am using the following command:

```
deepspeed examples/pytorch/translation/run_translation.py \
        --deepspeed tests/deepspeed/ds_config_zero3.json \
        --model_name_or_path facebook/m2m100_418M \
        --per_device_train_batch_size 8 \
        --per_device_eval_batch_size 8 \
        --output_dir output_dir --overwrite_output_dir \
        --fp16 \
        --do_train --do_eval --do_predict \
        --max_train_samples 500 --max_eval_samples 50 --max_predict_samples 50 \
        --num_train_epochs 0.001 \
        --dataset_name wmt16 --dataset_config "ro-en" \
        --source_lang en --target_lang ro \
        --predict_with_generate --forced_bos_token ro
```
    
Just to give you an example, if I train for 1 epoch I can get 20 BLEU points in the test set, but if I train for 3 epochs I get around 10 BLEU points.
Am I doing anything wrong? Does M2M100 requires any specific hyperparameter/hyperparameter configuration?
Thanks

### Expected behavior

I was expecting the model performance to improve for this specific language-pair (en-ro)
@patil-suraj could you help me with this problem?<|||||>Hi @joao-alves97, you should really ask those questions on the [forums](https://discuss.huggingface.co/) where the whole community will be able to help. We keep GitHub issues for bugs (with a reproducer) or feature request only :-)<|||||>I already did it @sgugger and no one answered me: https://discuss.huggingface.co/t/m2m100-training-does-not-improve-model-performance/23787 . And it seems to be a bug in my opinion. It is not normal that the loss continues decreasing but the performance does not improve at all.  Something must be wrong in the M2M100 implementation. I did an exhaustive hyperparameter search and it did not solve the issue.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.