I think you’re overthinking this really. from the point of view of pytorch, pytorch is procedural, rather than declarative. So, every step, every single step, you have to first run each operation. Like:
h1 = input @ W1 + b1
h2 = h1 @ W2 + b2
out = h2 @ W3 + b3

When you do this, you can freely choose what you’re going to calculate

the value of W1, b1, input etc is whatever they are at the time each line is executed. They’re just normal torch tensors more or less, with a couple of extra properties tagged on, but basically you can do what you like. If you switch one weight variable for another, the contents and dimensions of that variable are whatever you have assigned them to be previosuly.

Whenever you choose, you can take some gradOutput, and pass it to the output of your network:
out.backward(gradOutput)

It’ll then propagate backwards, and increment the gradients for your various W variables, and all the variables essentially, more or less.
Then, you’ll ask the optimizer to do something with these gradients (I think), but the optimizer will basically conceptually do something like add W.grad * learningRate to each W.
Changing tack a bit, there is a ton of original research, and research opportunities, around growing networks and so on. However, pytorch itself is not opinionated on this subject: it gives you the ability to calculate stuff, and run gradients through that stuff, but it wont magically grow your network, or transfer learning from one set of weights to another, and so on. You’ll need to write code to handle this. Or maybe write a library, so other people can re-use the same code.