螢幕快照 2020-08-24 下午7.42.141692×902 37.2 KB
The image is a simplified version of my network. I want to update Network1 with Loss_Network1 and Network2 with Loss_Network2. However, when I do backward(), the grads will be calculated for them regardless (ex: the grads for Loss_Network1 is added onto the .grad of Network2). The only approach I can think of now is:
Loss_Network1.backward()
optimizer_Network2.zero_grad()
copy the .grad for Network1
Loss_Network2.backward()
optimizer_Network1.zero_grad()
copy back the .grad for Network1
optimizer_Network1.step()
optimizer_Network2.step()
and also, doing optimizer_Network1.step() before Loss_Network2.backward() although may seem obvious but isn’t an option because that way it will raise: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Please let me know if there’s is a simple way to maybe freeze .grad in backward() so that I won’t need to copy the .grad s.    Thanks!!