I have never done that before, my guess would be to load the saved model and assign it to model.fc. Make sure that your connecting layer takes in model.fc.in_features.
Edit: In the use case of adding just a few extra linear layers, it really does not make any sense to have “trained linear layers” connected at the end because backbone is different to begin with. I do not see any benefits of doing this.