TF2 - how to access intermediate layers of pre-trained bert model?
(I'm following [this](https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/)  pytorch tutorial about BERT word embeddings, and in the tutorial the author is access the intermediate layers of the BERT model.)

What I want is to acess the last, lets say, 4 last layers of a single token input to the BERT model in tensorflow2 code. Because each layer output a vector of length 764 - so last 4 layers will have a shape of 4*768=3072 (for each token).

How can I implement in TF/keras/TF2, to get the intermediate layers of pretrained model for a token input? (later I will try to get the tokens for each token in a sentence, but for now one token is enough).

I'm using the huggingface's BERT model:

!pip install transformers
from transformers import (TFBertModel, BertTokenizer)

bert_model = TFBertModel.from_pretrained("bert-base-uncased")  # Automatically loads the config
bert_tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
sentence_marked = "hello"
tokenized_text = bert_tokenizer.tokenize(sentence_marked)
indexed_tokens = bert_tokenizer.convert_tokens_to_ids(tokenized_text)

print (indexed_tokens)
>> prints [7592]
The output is a token ([7592]), which should be the input of the for the BERT model
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
