The easiest way would be to load the complete model and manipulate the linear layer(s) afterwards.
Would that be an option? Otherwise you could probably filter out the unwanted parameters from your state_dict and load them using strict=False.