Indeed, upon rereading this paragraph:



 tom:

The way backpropagation works is to evaluate the gradients at the locations of the last forward pass.
In theory, you could also copy the network, make the parameters shared and then just add the loss to achieve the same. Then the backprop at real would go through one copy and the one at fake through the other.


seems to be misleading (but I cannot edit that post anymore), thank you for pointing that out.
Indeed, what doing the backward in two pieces saves is the saving two calculation graphs at the same time, this has nothing to do with the network (weights).
Best regards
Thomas