How to select a certain layer as token's representation?
My understanding from the paper that each token is represented by a 768-dim vector from the last hidden layer. Is it correct? If so, how can I get the second-to-the last layer's parameter as token representation? 

model = BertForTokenClassification.from_pretrained("bert-base-uncased", num_labels=len(tag2idx))
 list(model.named_parameters()) gives all weights but how can I know which parameters are from the second to the last layer?

for example, I have a word 'test', how can I get the BERT features from the second to the last hidden layer for the word 'test' in a given sentence?
Hi @yexing99,
What you need is the `hidden_state`, not the `weights`of the model.
Don't use `model.named_parameters()` but just use the output of the model.
Here is an example: https://github.com/huggingface/pytorch-pretrained-BERT#bert
And more details here: https://github.com/huggingface/pytorch-pretrained-BERT#1-bertmodel
You should do somethings like this to get the BERT features from the second to the last hidden layer if the word `test` is for example the *fifth* word in your sentence (index 4 in the sentence):
```python
encoded_layers, _ = model(tokens_tensor, segments_tensors)
features = [encoded_layer[4] for encoded_layer in encoded_layers[1:]]
```
