wav2vec 2.0 loaded from a fairseq checkpoint isn't strictly equivalent
## Environment info
- `transformers` version: 4.9.2
- Platform: Linux-4.18.0-147.51.2.el8_1.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1 
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

## Who can help
@patrickvonplaten @anton-l 

## Information

This is related to wav2vec 2.0 loaded with Wav2Vec2Model

The problem arises when using:
* [x] my own modified scripts: (give details below)

## To reproduce
Steps to reproduce the behavior:

1. train a base model with fairseq (official base config) and convert the checkpoint with the official HF script.
2. Load the converted model with Wav2Vec2Model
3. Load the fairseq checkpoint with fairseq.checkpoint_utils.load_model_ensemble_and_task
4. Print a sum of all parameters and the output of identical input wav file.

## Expected behavior

In practice, we should see two identical sum as the models should be exactly identical. This is not the case in my test. Then, if you pass the same wav file in the two models (HF and fairseq), you'll get different outputs as well. 

What I did to compute the sum:
![image](https://user-images.githubusercontent.com/11910731/134816102-8bd233af-0666-42e3-aae8-d71f2a3572b8.png)

The output:
![image](https://user-images.githubusercontent.com/11910731/134816107-291c9a9d-2d21-4f53-9215-af217a5ea0bb.png)

As you can see, we have the same number of parameters, but not the same values as they differ slightly. 

Then, I let both models train on the exact same downstream task (everything runs within SpeechBrain). The downstream setups are **identical**. At the end, the fairseq models as a WER 2% lower while we should see the exact same results.

Hey @TParcollet,

Thanks for the issue. Could you try to make a fully reproducible code snippet? I image that one wouldn't have to train a whole new model, but could also just use one of the already pretrained wav2vec2 models from fairseq to see this difference no?

Also it would be great if you could replace the screenshots by code snippets which would make it easier to reproduce the error for us. Thanks!<|||||>I'll do this as soon as I have 5 minutes of spare time :-P <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.