Converting TFBartForConditionalGeneration to BartForConditionalGeneration does not work
### System Info

Transformers version: 4.22.1
Platform: SMP Debian 4.19.249-2 (2022-06-30) x86_64 GNU/Linux
Python version: 3.7.12

### Who can help?

@patil-suraj

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import BartConfig, BartForConditionalGeneration, TFBartForConditionalGeneration

config = BartConfig(vocab_size = 32001,
                    pad_token_id=0,
                    bos_token_id=2,
                    eos_token_id=3,
                    is_encoder_decoder=True,
                    decoder_start_token_id=2,
                    forced_eos_token_id=3,
                    max_position_embeddings = 1024,
                    d_model = 768,
                    encoder_layers = 6,
                    encoder_ffn_dim= 768 * 4,
                    encoder_attention_heads=12,
                    decoder_layers=6,
                    decoder_ffn_dim=768 * 4,
                    decoder_attention_heads=12)

#### Init and Build TFBart
tf_model = TFBartForConditionalGeneration(config)
tf_model.build(input_shape = (None, None)) # Without this, tf weights are not initialized

#### Save Tensorflow Model
tf_model.save_pretrained('tf_model')
weights = [(tf_model.weights[idx].name, tf_model.weights[idx].numpy()) for idx in range(len(tf_model.weights))]

#### Load Tensorflow Model into PyTorch Model
pt_model = BartForConditionalGeneration.from_pretrained('tf_model', from_tf=True)
params = [(name, param.detach().numpy()) for name, param in pt_model.named_parameters()]

### Expected behavior

I build a TFBart, save its weights, and then load its weights into PTBart.
When I debug and inspect weights (tf) and params(pt), only the first layer (shared embedding) weights are correctly loaded.
However, when I initialize a PT model and load its weights to TF one, all of the weights are correctly loaded.

I thought the issue could be the input_shape when building tf model so I tried various values such as 1024, 1026, and such with no result.

I also want to be able to set these weights manually (almost one-by-one) from a numpy matrix. The use case is to train a BART model from scratch using custom tensorflow model and load its weights into huggingface's BART model.
So what is the method to set weights manually for both TF and PT models?
Hi @meliksahturker 👋  

Loading TF weights into a PT model is an uncommon task, so our codebase may have issues. I'd like to request two things:
1. I've recently touched TFBart code -- can you confirm that the problem persists with the current `main` (`pip install git+https://github.com/huggingface/transformers.git`)?
2. If running from `main` doesn't solve it, can you share the code you're using to compare the weights?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.