Using BERT as feature extractor
Following file extract_features.py, I use bert-large-uncased and it outputs 4 layer outputs for each token(word). Since I want to use it as feature extractor of entire sentence, which values should I use? Or  is there any other processing we should do(like concatenate output for last token)?
i have same question <|||||>How best to fine-tune or pool BERT is an open question in bertology :p 

["How to Fine-Tune BERT for Text Classification?"](https://arxiv.org/pdf/1905.05583.pdf) has a comprehensive overview. Look at table 3 specifically which found that taking the max of the last 4 layers achieves the best performance (in their testing configuration with some fine tuning), this would be my guess. 

You can also look at the approaches taken by [bert-as-a-service](https://github.com/hanxiao/bert-as-service#q-what-are-the-available-pooling-strategies) which are explained pretty well, in this repo they use MEAN as the default pooling strategy.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
