From the documentation you see that the PyTorch developers have given serialization quite a bit of thought and arrived at the conclusion that - as you discovered yourself - it is too fragile. That is why they adopted the approach of providing factory functions e.g. in torchvision.
So what you would need to do to follow that best practice is to put your models in a module that Python then can load. This isn’t PyTorch specific, but a general limitation of Python serialisation.
In the meantime, you could see if JITed modules work for you. They intentionally store the parameters and computation in a file so you can use it without further dependencies. The obvious drawback is that - to work around Python’s serialisation limitations - Python is left out of the loop, so the saved model is not the same as a model loaded from a Python module.
Best regards
Thomas