feature-extraction pipeline is second last layer better representation than the last hidden layer
# 🐛 Bug

I see that for feature-extraction pipeline, you output the last hidden layer of the transformer.  Another approach I have seen is people using second to last layer as the output (Bert-as-a-service).  The rationale being that the last layer is too close to the model task such as masked language model objective. I am a bit confused as to what would be an ideal representation layer for sentence embedding.

Model I am using (Bert, XLNet ...): Bert


## Environment info

     
- `transformers` version: 2.8.0
- Platform: CPU
- Python version: 3.7
- PyTorch version (GPU?): 1.5.0
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

This is quite a general question, and I fear there is no one right answer. You can see the "best layer to use" as another hyperparameter to tune in your task: the right value will depend on your task, dataset, and query. Oftentimes paper do some kind of evaluation of this. For instance, in the original [BERT paper](https://arxiv.org/pdf/1810.04805.pdf) they use BERT for feature extraction and evaluate which layer combinations worked best (Table 7). They find that the second-to-last layer performs better than the last layer, but that a concatenation of the last four layers is best. But again, that is using BERT, on a specific task, with a specific dataset. Your results may differ, so it's best to test and tune to your problem.<|||||>@BramVanroy @kaushaltrivedi How do you customize what layers go into the feature representation (via the pipeline)?