You could just load the state_dict into your model, probably set it to evaluation with model.eval(), and change the forward pass to return your desired activation.
To save and load a model, have a look at the Serialization semantics 4.