save all 12 layers outputs for each token
Hi,

I am following [this](https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/) post to create sentence vectors from the last 4 layers or sum all 12 layers and so on.

Considering a given document has 10,000 tokens and I am planning to use  "bert_base_uncased", I was wondering how to save all 12 layers for each token efficiently. Also, further, how to retrieve these layers in order to build sentence vectors.

Thanks!
Hi, to retrieve all layers you can set the `output_hidden_states` flag to `True` in your model configuration:

```py
from transformers import BertConfig, BertModel

config = BertConfig.from_pretrained("bert-base-uncased")
config.output_hidden_states = True
model = BertModel.from_pretrained("bert-base-uncased", config=config)

outputs = model(inputs)
hidden_states = outputs[-1]
```

The model will now output an additional value which will be a tuple of size 1 + n_layer:
`hidden_states[0]`: output of the embedding layer
`hidden_states[1]`: output of the first layer
...
`hidden_states[12]`: output of the last layer
