Thanks for the code snippet.
The same root cause can be seen in your code, as output2 and output3 share self.fc1, which is also the reason why you would need to use loss3.backward(retain_graph=True).
optimizer3.step() will update the parameters of fc3, which would then compute a wrong gradient during the the backpropagation to fc1.