Is there any plan of developing softmax-weight function for using 12 hidden BERT layer? 
Thanks for developing very nice/useful library.

My question is about using 12/(or in large model, more) hidden layer.
First, Does how to use hidden layers depend on the downstream task?
(Say, concat, average, only final layer, only mean of top 4 layer, etc...)

For using all layer, I think it's good to use softmax weight. During training , hidden layer's feature is fix but weight is learned for the task. So second question is, Is there any plan of developing softmax-weight function for using 12 hidden BERT layer? 

Thanks
Yes we might add a module for scalar mixture of layers like the one of AllenNLP, for instance (https://github.com/allenai/allennlp/blob/master/allennlp/modules/scalar_mix.py).<|||||>I'm looking forward to see that in also pytorch-transformer.
Again, thanks! I'll keep track on this repository.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
