I’ve not quite got to that stage yet but I think it’s normal? My understanding is you pre train say XLMRobertaForMaskedLM which creates a model consisting of an encoder with an MLM head and no pooling layer. Once that’s trained you load it as a XLMRobertaForSequenceClassification which copies the trained encoder and adds pooling and a sequence classifier - you then need to fine-tune this?