Thank you for your fast answer! The optimizer3 only updates the parameters of part3, therefore the new parameters should not affect loss2 (only depends from parameters of part 1 and 2).
If I got it right, in the post you mentioned the solution would be to set retain_graph=False? This won’t work for me, since it results in the following error

RuntimeError: Trying to backward through the graph a second time (or directly access saved variables after they have already been freed).

In this case, I think autograd is trying to backpropagate through part 3 a second time (which would be as I want it to). Even if I calculate the loss3 again after the gradient step of optimizer3 (to get the loss dependent from the new parameters) I am getting this error when calling loss12.backward().
Am I misunderstanding an important concept here or where is my problem?
Thank you!