Indeed, I don’t need to retain the graph in this case but still confuse why memory keep growing.
For each training step, this  loss.backward(retain_graph=True) retain the graph and next
 output = G(im_data_tu) 
 loss_t = adentropy(F1, output, 0.1)
 loss_t.backward()

create another graph and these should be the maximum usage of memory. Did increased memory come from the retained graph ? I did comment out output = G(im_data_tu)   and memory usage seem normal. Could you explain that ? Thanks !