What should i do if I want a model class similar to BertForSequenceClassification?
# ❓ Questions & Help

Hi!
For my task I want to implement a class similar to BertForSequenceClassification, but a little different, for example, i may want multiple linear head on top of the bert encoding. Call it BertForMySequenceClassification,

But I still want to use the pretrained bert parameters, and hopefully do easy load/save.
For example, i hope i can BertForSequenceClassification.from_pretrianed(some_path)

Any suggestion or help are much appreciated! 
Thanks!
Hi @cloudygoose , 
Take a look at the code of  `BertForSequenceClassification` https://huggingface.co/transformers/_modules/transformers/modeling_bert.html#BertForSequenceClassification,

All it does is, pass the inputs through `BertModel`, which returns the pooled output of bert and the applies the classifier on top it.

You can follow the same process, create `YourSequenceClassification` class by sub-classing `BertPreTrainedModel`,
init your `BertModel` and the additional linear heads and final classfication layer and do forward as  
`BertModel -> additional linear heads -> classifier`

You can just take the `BertForSequenceClassification`  as is and additional layers before the `classifier` 

Hope this helps<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
