Issue with Fairseq BART checkpoint conversion in Fairseq 0.12
Hi, I pretrained a BART model using Fairseq 0.12.2. Then, I tried to convert it using the official script, https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/convert_bart_original_pytorch_checkpoint_to_pytorch.py, and it didn't work out of the box due to unexpected and missing keys in the state dict. Changes I have introduced:

1. Rename keys with `_weight` to `.weight`, and the same with bias. Ok, no doubts here.
2. `decoder.output_projection.weight` is not present in HF's model. Check that `torch.isclose(state_dict["decoder.embed_tokens.weight"], state_dict["decoder.output_projection.weight"])`, and then delete `decoder.output_projection.weight` (it seems to be an additional shared weight present in Fairseq's dictionary). I think it's ok but not 100% sure.
3. Now, the most problematic part: Fairseq's BART has 4 additional parameter sets for each layer: `encoder.layers.0.in_proj_weight, encoder.layers.0.in_proj_bias, encoder.layers.0.out_proj_weight, encoder.layers.0.out_proj_bias`. They are declared here: https://github.com/facebookresearch/fairseq/blob/v0.12.2/fairseq/modules/transformer_layer.py#L75. If I'm not wrong, they are only used if BT (BetterTransformer, check: https://pytorch.org/blog/a-better-transformer-for-fast-transformer-encoder-inference/) is being used: https://github.com/facebookresearch/fairseq/blob/v0.12.2/fairseq/modules/transformer_layer.py#L308. Apparently, it's not in my case (because BT is only supported with torch >=1.12). So, if I understand correctly, I can safely delete these keys, which I did.

After these modifications, the state dict can be loaded. The assertion `assert fairseq_output.shape == new_model_outputs.shape` is ok, but the assertion `assert (fairseq_output == new_model_outputs).all().item()` isn't. The outputs seem to be similar enough at first glance:

```
>> fairseq_output
tensor([[[-1.8069, -0.4117, -0.9243,  ...,  0.6273, -1.3608,  0.1720],
         [ 2.0357,  1.1873, -1.8202,  ...,  0.8068, -1.3662,  0.1693],
         [ 5.2753, -0.7177,  0.6229,  ...,  0.0101, -1.6409,  1.5729],
         ...,
         [-1.0434,  3.7756, -3.0025,  ..., -3.6483,  1.4942,  4.0406],
         [-1.1989,  4.8781, -2.4865,  ..., -4.0379,  0.7374,  3.6970],
         [-1.0135,  4.2036, -2.6940,  ..., -3.5853,  1.5794,  4.0704]]])
>> new_model_outputs
tensor([[[-1.8047, -0.4085, -0.9251,  ...,  0.6256, -1.3601,  0.1714],
         [ 2.0344,  1.1864, -1.8195,  ...,  0.8076, -1.3673,  0.1676],
         [ 5.2745, -0.7172,  0.6230,  ...,  0.0093, -1.6411,  1.5726],
         ...,
         [-1.0617,  3.6596, -2.9122,  ..., -3.6041,  1.4983,  4.0248],
         [-1.2374,  4.8001, -2.4190,  ..., -3.9873,  0.7194,  3.6830],
         [-1.0299,  4.0773, -2.6344,  ..., -3.5592,  1.6123,  4.0465]]])
```


But using `torch.isclose` says otherwise, even if setting the default epsilon to very high values. Eg.:

```
torch.isclose(fairseq_output, new_model_outputs, atol=1e-01)
tensor([[[ True,  True,  True,  ...,  True,  True,  True],
         [ True,  True,  True,  ...,  True,  True,  True],
         [ True,  True,  True,  ...,  True,  True,  True],
         ...,
         [ True, False,  True,  ...,  True,  True,  True],
         [ True,  True,  True,  ...,  True,  True,  True],
         [ True, False,  True,  ...,  True,  True,  True]]])
```

Trying to generate tokens with BartForConditionalGeneration seems to be leading to gibberish. Any help will be much appreciated, thanks in advance!

# System Info

```
transformers '4.22.2'
torch '1.10.2+cu113'
fairseq '0.12.2'

```

Tagging @patil-suraj

Apparently, the issue was related to the tokenizer, and now all assertions hold. Meaning that the changes I made do work, in case someone is interested. Closing the issue.