I meant to say, that it might not always be feasible or desired to inherit from PreTrainedModel class. As you’re suggesting inheriting from GPT2PreTrainedModel is too specific. It would be good to have compatibility for nn.Module.