TFPreTrainedModel.build breaks pytorch PreTrainedModel.from_pretrained(from_tf=True)
### System Info

- `transformers` version: 4.30.2
- Platform: macOS-13.4-x86_64-i386-64bit
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): 2.10.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help?

@Rocketknight1 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The `build()` overridden method introduced recently breaks `PreTrainedModel.from_pretrained(from_tf=True)`. This can be reproduced by the following line of codes:

```python
model = RobertaModelForSequenceClassification.from_pretrained(path, from_tf=True)
```
console prints out:
```
All TF 2.0 model weights were used when initializing RobertaForSequenceClassification.

Some weights of RobertaForSequenceClassification were not initialized from the TF 2.0 model and are newly initialized: ['roberta.embeddings.word_embeddings.weight', 'roberta.embeddings.position_embeddings.weight', 'roberta.embeddings.token_type_embeddings.weight', ... all 6 layers... 
```

After some digging, I believe the recursive call to `__call__` in the `build()` causes the names of all TF weights to be prefixed twice with the model's keras name (instead of once,) e.g. `tf_roberta_for_sequence_classification/tf_roberta_for_sequence_classification/...`. The `from_pretrained(from_tf=True)` works according to the following steps:

1. create a PT model
2. create a TF model
3. build the TF model by calling `tf_model(tf_mode.dummy_inputs, training=False)`
    1. calling `tf_model._call_` 
    2. enter name scope of the model, e.g. `tf_roberta_for_sequence_classification`
    3. figure it has not been built because `built=False` 
    4. call `tf_model.build` 
    5. the overridden `TFPreTrainedModel.build` then set `built=True` 
    6. calls `self._call__` (i.e. `tf_model.__call__`) again
    7. **enter name scope of the model again**, e.g. e.g. => `tf_roberta_for_sequence_classification/tf_roberta_for_sequence_classification`
    8. proceed to call the `tf_model.call`
    9. call the layer’s `_call_` 
    10. add variables with name e.g. `tf_roberta_for_sequence_classification/tf_roberta_for_sequence_classification/roberta/...`
4. load TF weights to TF models
5. map TF weight names to PT weight names by removing the **first** prefix of the TF variable names and copy over the weights. => fail to map TF names to PT names due to the double prefix e.g. => `tf_roberta_for_sequence_classification/tf_roberta_for_sequence_classification`

The hacky workaround is to override the `build` ourself to not call the `__call__`, e.g.:

```python
class _FixedTFRobertaForSequenceClassification(transformers.TFRobertaForSequenceClassification):

    def build(self, input_shape=None):
        self.built = True

if id(transformers.TFRobertaForSequenceClassification) != id(_FixedTFRobertaForSequenceClassification):
    print('fixing TFRobertaForSequenceClassification to', _FixedTFRobertaForSequenceClassification.__name__)
    transformers.TFRobertaForSequenceClassification = _FixedTFRobertaForSequenceClassification
model = RobertaForSequenceClassification.from_pretrained(language_model_path, from_tf=True)

#### console prints out
All TF 2.0 model weights were used when initializing RobertaForSequenceClassification.

All the weights of RobertaForSequenceClassification were initialized from the TF 2.0 model
```

The double naming can also be seen by just creating the submodel:
```python
config = RobertaConfig.from_pretrained(language_model_path)
tf_model = TFRobertaForSequenceClassification(config)
print(tf_model._name)
tf_model.weights[10]
# => notice the double prefix in the weight variable names
```

There should be better fix.

CC @Rocketknight1 

### Expected behavior

1. TF submodel variables/weights names should not be double-prefixed
2. `from_pretrained(from_tf=True)` should works.
@winston-zillow thanks for the bug report! We're investigating now.<|||||>Hi @winston-zillow, I did some investigation and I can't reproduce the issue! At my end `RobertaModel.from_pretrained("roberta-base", from_tf=True)` or `RobertaForSequenceClassification.from_pretrained("roberta-base", from_tf=True)` both work correctly.

This might have something to do with the specific checkpoint you're using. Can you give me some code that reproduces the issue using a checkpoint on the HuggingFace Hub, or if not, can you upload the checkpoint you're using so I can try to figure this one out?<|||||>Wait, I was able to trigger the bug by switching my TensorFlow version! Investigating this now.<|||||>Update: The bug is not caused by `build()`, but by faulty imports from the deprecated `tf.python.keras` repo. As a workaround for now, you can update your version of TensorFlow to 2.11 or newer, which should solve the bug for you. I'm working on a PR which should fix this issue for all TF versions >= 2.6, and bump our minimum supported TF version to 2.6 as well.<|||||>@winston-zillow A PR is in that should resolve this! If you want to try it before it's merged and report your experiences, you can use
```
pip install git+https://github.com/huggingface/transformers.git@improved_keras_imports
```<|||||>PR has now been merged! You can now get it just by installing from `main`:
```
pip install git+https://github.com/huggingface/transformers.git
```
It'll also be included in the next release of transformers. Thanks again for filing the issue, and please feel free to comment or reopen it if the PR doesn't resolve your problem! Our test suite normally catches things like this, but the specific combination of older TF versions and TF -> PT crossloading slipped through, so the bug report is greatly appreciated.<|||||>@Rocketknight1  Thanks for the quick fix! 