title: How to get intermediate layers' output of pre-trained BERT model in HuggingFace Transformers library?
body: <p>(I'm following <a href="https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/" rel="noreferrer">this</a> pytorch tutorial about BERT word embeddings, and in the tutorial the author is access the intermediate layers of the BERT model.)</p>
  
  <p>What I want is to access the last, lets say, 4 last layers of a single input token of the BERT model in TensorFlow2 using HuggingFace's Transformers library. Because each layer outputs a vector of length 768, so the last 4 layers will have a shape of <code>4*768=3072</code> (for each token).</p>
  
  <p>How can I implement this in TF/keras/TF2, to get the intermediate layers of pretrained model for an input token? (later I will try to get the tokens for each token in a sentence, but for now one token is enough).</p>
  
  <p>I'm using the HuggingFace's BERT model:</p>
  
  <pre><code>!pip install transformers
  from transformers import (TFBertModel, BertTokenizer)
  
  bert_model = TFBertModel.from_pretrained("bert-base-uncased")  # Automatically loads the config
  bert_tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
  sentence_marked = "hello"
  tokenized_text = bert_tokenizer.tokenize(sentence_marked)
  indexed_tokens = bert_tokenizer.convert_tokens_to_ids(tokenized_text)
  
  print (indexed_tokens)
  &gt;&gt; prints [7592]
  </code></pre>
  
  <p>The output is a token (<code>[7592]</code>), which should be the input of the for the BERT model.</p>