CUDA memory continuously increases when net(images) called in every iteration
Kalamaya
This is because pytorch will build a the graph again and again, and all the intermediate states will be stored.
ruotianluo
apaszke
I think that did it! Ugh! I spent two days on this! >< Thanks though… so what was going on here?
Kalamaya
<SCODE>for i in xrange(100):
    out = net(input) 
    loss = someLossFunction(out)
<ECODE>
Kalamaya
The graph won’t be re-made. The 100 loops basically create 100 graphs. The 100 graphs share the same input and parameters, but all the intermediate variables of 100 graphs (although they could be the same) are saved separately.
ruotianluo
Kalamaya
<SCODE>for i in xrange(100):
    out = net(input)
    loss += someLossFunction(input) # BAD, because it keeps continuing the graph over the for-loop

    loss = someLossFunction(input)   # this is fine

    loss = someLossFunction(input)
    total_loss += loss.data[0]              # this is fine
<ECODE>
smth
Im just asking for the sake of understanding: Is what is happening, let me put it differently: Are you saying that this statement here, will make two graphs that are identical to each other? loss = someLossFunction(input1) + someLossFunction(input2) Is my conclusion correct?
Kalamaya
yes, your conclusion is correct.
smth
To clarify for RNN users that might come across this, we do actually want to keep the graph around for backprop-through-time. (Right? Or is there a better way?)
spro
smth
This should be written in the tutorial!
heihei
hmishfaq
hi, can the total_loss here be back propagated?
11112
this solved my problem
Felix_Lessange
This solved my problem too. I didn’t know call loss instead of loss.data[0] will cause rewriting the graph.
ddeng
This is really important. Everyone learning PyTorch should know this at the first place.
zhaoethz
Albert_Christianto