Inspect inner layers of Transformer models as in TensorFlow/Keras
In `keras` (and `tensorflow`), we can inspect a model as follows:

```
resnet = tensorflow.keras.applications.ResNet152V2(
    include_top=True,
    weights="imagenet",
    input_tensor=None,
    input_shape=None,
    pooling=None,
    classes=1000,
    classifier_activation="softmax",
)

resnet.summary()
```

This will print a handy summary of the model's inner layers:

```
Model: "resnet152v2"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 224, 224, 3) 0                                            
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D)       (None, 230, 230, 3)  0           input_1[0][0]                    
__________________________________________________________________________________________________
conv1_conv (Conv2D)             (None, 112, 112, 64) 9472        conv1_pad[0][0]                  
__________________________________________________________________________________________________
pool1_pad (ZeroPadding2D)       (None, 114, 114, 64) 0           conv1_conv[0][0]
```

When instantiating a Transformer model, such as in the following snippet

```
from transformers import TFAutoModelForSequenceClassification
checkpoint = "bert-base-uncased"
model = TFAutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
```

the same operation (`model.summary()`) prints reduced information:
```
Layer (type)                 Output Shape              Param #   
=================================================================
bert (TFBertMainLayer)       multiple                  109482240 
_________________________________________________________________
dropout_75 (Dropout)         multiple                  0         
_________________________________________________________________
classifier (Dense)           multiple                  1538      
=================================================================
```

I'd like to see "inside" the `bert` block and work with the model as with any `keras` model.
This means iterating over the layers, getting the weights, etc.  I am aware that we can approach this by saying, e.g.,

```
for w in model.weights:
    print(w.name, w.shape)
```

However, is there any way to see the inner layers (the layers inside the `bert` block above) or, in general, use the Transformer model like a regular `TensorFlow` model? An example of what I want to "see" is in [this](https://keras.io/examples/nlp/text_classification_with_transformer/) pure-TensorFlow example.





cc @Rocketknight1  @gante <|||||>Assigning to me, I can see if there is good solution for this<|||||>This is tricky, I think. We spoke briefly about this with fchollet - I don't think there's a good way to get this without refactoring our models from subclassing-style to functional-style, but I might be wrong!<|||||>Yup, was reaching the same conclusion. Enabling a clean `model.summary()` would imply a massive refactor in all models, such that everything is under the `Functional` model framework 👀 

@phrasenmaeher, we're sorry that we are not much help, but it would imply a massive change for a minor benefit. I'm closing the issue, but please, if you think there is another way to solve the question, feel free to reopen :)