Cysu:

calling backward twice is recommended as it could save some GPU memory


sorry, but may i ask why does it save memory?



 Cysu:

The problem mainly attributes to reusing the same label tensor


Another question, do you mean the code in referenced link (L219,L232) is wrong because it can cause the problem (different result between twice backward and once backward)? But why…? why same label tensor can cause this difference?
The last thing, more important, i didn’t use any inplace operations in my code like label.fill_(fake_label), but i also encountered the problem that:
the training result of twice backward is different from add the loss together then use 1 backward…
the later one will cause nan gradient and nan loss in the training, i don’t know why…
every time, i input 1 pair (2 different input with same label) to my network, to be clear:
# method 1
out1 = model(input1)
loss1 = criterion(out1,label)
loss1.backward()
optimizer.step()
optimizer.zero_grad()

out2 = model(input2)
loss2 = criterion(out2,label)
loss2.backward()
optimizer.step()
optimizer.zero_grad()

# method 2
out1 = model(input1)
out2 = model(input2)
loss1 = criterion(out1,label)
loss2 = criterion(out2,label)
lossAdd = loss1 + loss2
lossAdd.backward()
optimizer.step()
optimizer.zero_grad()

in my experiment, method 1 and method 2 have different results in training, and method 2 can cause Nan problems after several training epochs (training can not converge). May i ask why does this happen?
is the reason that i used same label tesor?
is because input1 and input2 are too different from each other so can not add their loss together?
Very look forward to your reply…
@han @Nabarun_Goswami  @tymokvo @tom @Cysu @SimonW @Sten_Sootla @Weifeng @Atcold @yingda.yin
sorry to interrupt so many people with so old post, but i really got stuck here for more than one month, really really need help here…