Flaky test for NLLB-MoE Model 
### System Info

Environment: Circle CI test_torch image, defined [here](https://github.com/huggingface/transformers/blob/c15f937581048fa28f77794c4f6a257e14d272cb/.circleci/create_circleci_config.py#L211)

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The test `tests/models/nllb_moe/test_modeling_nllb_moe.py::NllbMoeModelTest::test_decoder_model_past_with_large_inputs` is flaky and occasionally fails. Note: the tolerance on the test is also very low. 

Error cannot be deterministically reproduced. The test passes when I run it locally. 

An example of a failed run: https://app.circleci.com/pipelines/github/huggingface/transformers/60931/workflows/4642cd50-8d8c-4a6b-bea9-b077ff7400cf/jobs/747651

This came from this run: https://github.com/huggingface/transformers/runs/12389736536

And this unrelated PR: https://github.com/huggingface/transformers/pull/21855  

### Expected behavior

Tests pass on every run. 
Yep, I have no idea why 😓 <|||||>Looked into this and I think the flakiness is caused by the natural variability in the sparse MoE layers. Specifically that when they calculate which experts to use in the gating logic, they’re computing probabilities imperfectly for two different sets of inputs: one with prior inputs concatenated with the past key values and one with just the past key values. 

The test usually passes cause magnitude of the difference is usually likely to be small. Notably, when the vocab size is increased this pass rate goes up (and vice versa) since the increased representational capacity can help the model make more accurate decisions about which experts to use for each input. For example, increasing the vocab size in the config from its current 99 to 999 increases the pass rate from ~80% to ~95%.

I think this flakiness is inherent in the sparse layers, but if I understand right the point of the test is to check the decoder uses the past properly, so I put up a PR that edits the test to use dense layers and moves to the rtol down to 1e-3 to be in line with the other models’ version of this check.