I have a large text corpus, and a small subset of it that is labelled for a mutli-label text classification task. I’ve seen many (excellent!) examples of fine-tuning different models for sequence classification, but I couldn’t find one in which the training is separated to two distinct stages:

Fine-tune a specific language model on a specific corpus (unsupervised).
Train a sequence classification model on top of it on a labelled subset of the original dataset (supervised).

I assume this is a pretty common (and simple) scenario, but I couldn’t find any relevant docs. If anyone can provide any pointers, it would be much appreciated!