How much does memory increase? Did you try with 3 separate checkpoint files, with different weights?
In any case, I’m not familiar with the internals of PyTorch, these are mainly exploratory questions so that I can learn more about PyTorch thanks to your situation  It just seems logical that there would be some initialization at some point for such a complex framework.