Best practices for combining large pretrained models with smaller models?
Hello,

If I were to try to combine a large model (like BERT) with a smaller model (some variation of fully connected, convolutional network with significantly less params and pretraining) by jointly training them and concatenating their outputs for a final classifier, what would be some things I should consider?

For example, should they have different optimizers and learning rates? Should I try to keep the number of params in the smaller model relatively small? What would be some good ways of fusing the output of BERT and the output of the small model besides concatenating? 

I'd really appreciate any insight from anyone who's tried something like this or have thought about it. Thank you!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@dchang56 Any updates? I am looking to do this as well. <|||||>Hi! I see that you're also doing scientific/medical NLP :)
I sent you an email at your gmail address.