Approaches for semi-supervised fine-tuning after self-supervised pre-training My understanding is that self-supervised learning approaches approximately work like the following (I have Wav2Vec 2 in my mind here, used in speech recognition, but NLP transformer models are similar):

*

*You are training an "context representation" or "embedding" function $f_\theta$ minimizing (some loss involving) a contrastive loss on unlabelled data.  This happens at a pre-training phase with data not from your domain, and $f_\theta$ a huge model you get from somewhere (Huggingface etc.).

*You fine-tune a classifier $g_\phi$, taking as input the context representation, on labelled data $D$, with $f_\theta$ frozen, minimizing a prediction loss (cross entropy etc.). $D$ is from your domain, but much smaller than in pre-training.

*Predictions are made using $g_\phi \circ f_\theta$.

Now suppose you want to do semi-supervised learning: your domain data $D$ consists of a labelled part, plus a lot of unlabelled observations.  The next best thing would be to modify fine-tuning in (2) to use some off-the-shelf semi-supervised method (e.g., pseudo-labelling) and train $g_\phi$ utilizing all the labelled and unlabelled data.
My question: is it not also a sensible approach instead to follow both steps (1) and (2), and fine-tune $f_\theta$ using the unlabelled part of $D$, and then $g_\theta$ using the labelled part?  Combining more contrastive learning and regular supervised learning instead of switching to a different method?
I have not been able to find any work which investigates this setting (it is hard to search for -- "semi-supervised" just leads to false positives talking about the self-supervision).  Thinking about it, it feels like the more straigh-forward solution, instead of a new semi-supervised method; on the other hand, I would not be suprised if it was futile, as the new information provided to the contextual representation is very small to whatever was available in pre-training, leading to the unlabelled part contributing almost nothing.

Yes, it is indeed a sensible approach and has been done before in e.g., ULMFiT and DAPT. Substantial improvements are observed in both cases.
