[TFMarian] Slow integration tests are failing
After having uploaded the TF weights of: 
- https://huggingface.co/Helsinki-NLP/opus-mt-mt-en/commit/552db365bf294f7a2604fadcedfca0ed5b29bd66
- https://huggingface.co/Helsinki-NLP/opus-mt-en-zh/commit/137ef1a50f7a0eaf22a7d5685d07b66bb670ddd1
- https://huggingface.co/Helsinki-NLP/opus-mt-en-ROMANCE/commit/1854185e5a3183d8c73360b1cd53f63c2fb0ed46

and merged this PR: 

https://github.com/huggingface/transformers/pull/10664

the TFMarian slow integration tests are falling. This doesn't seem to be an easy issue and needs further investigation.

cc @patrickvonplaten @patil-suraj 
This is a very weird bug actually and can be reproduced the easiest ass follows:

```python
#!/usr/bin/env python3
from transformers import TFMarianMTModel, MarianMTModel, MarianTokenizer

tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-mt-en")

model_pt = MarianMTModel.from_pretrained("Helsinki-NLP/opus-mt-mt-en")
model_tf = TFMarianMTModel.from_pretrained("Helsinki-NLP/opus-mt-mt-en", from_pt=True)

model_tf.save_pretrained("./")
model_tf = TFMarianMTModel.from_pretrained("./")

input_str = "My name is Wolfgang and I live in Berlin"
input_str = "Billi messu b'mod ġentili, Ġesù fejjaq raġel li kien milqut bil - marda kerha tal - ġdiem."


output_tokens_pt = model_pt.generate(tokenizer(input_str, return_tensors="pt").input_ids)
output_tokens_tf = model_tf.generate(tokenizer(input_str, return_tensors="tf").input_ids)

print("Pt:", tokenizer.batch_decode(output_tokens_pt))
print("Tf:", tokenizer.batch_decode(output_tokens_tf))
```

fails while commenting out the save and load lines works:

```python
#!/usr/bin/env python3
from transformers import TFMarianMTModel, MarianMTModel, MarianTokenizer

tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-mt-en")

model_pt = MarianMTModel.from_pretrained("Helsinki-NLP/opus-mt-mt-en")
model_tf = TFMarianMTModel.from_pretrained("Helsinki-NLP/opus-mt-mt-en", from_pt=True)

input_str = "My name is Wolfgang and I live in Berlin"
input_str = "Billi messu b'mod ġentili, Ġesù fejjaq raġel li kien milqut bil - marda kerha tal - ġdiem."


output_tokens_pt = model_pt.generate(tokenizer(input_str, return_tensors="pt").input_ids)
output_tokens_tf = model_tf.generate(tokenizer(input_str, return_tensors="tf").input_ids)

print("Pt:", tokenizer.batch_decode(output_tokens_pt))
print("Tf:", tokenizer.batch_decode(output_tokens_tf))
```

Most other TFMarian models work correctly. This is pretty weird though and will need more time for investigation (cc @patil-suraj for info)<|||||>Thanks for your investigation!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Unstale<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.