@cakiki conceptually, I wonder if training my own language model and then fine-tune it for text-classification will work better than fine-tuning the same old distilbert model that everybody is using. The corpus I am working on is highly specialized (say, medicine for instance) so a dedicated language model makes sense.
What do you think?