Hi @adamh, did you find out how to do this? I also want to leverage a fine-tuned language model for classification. As far as I can tell when we choose a classification head then we must discard the previous head - i.e. in this case our beautifully fine-tuned LM head! So, can we put a classification head on top of this rather than discard it, or - do we need to unlock the next layers of our LM and propagate the weights down into the model so that when discarding our LM head we still have an underlying model that has inherited our fine-tuning?