Basically retain_graph option retains the intermediate tensors created while construction of computation graph. Without this, backward function would destroy all the connected computation graph.

  
      stackoverflow.com
  
  
      
    
  

  What does the parameter retain_graph mean in the Variable's backward() method? 7



  neural-network, conv-neural-network, backpropagation, pytorch



  asked by
  
  
    jvans
  
  on 04:11PM - 16 Oct 17


  
  
    
    
  
  


Picture in above question could be helpful.
In this case, what we want to retain is the intermediate tensors created from
feature_target = model(input1)
feature_identity = model(input2)

part, which should be used to compute gradients later.
If you call loss.backward() without retain_graph option, you will lose information about not only feature_adversarial of i-th iteration, but also feature_target and feature_identity since loss computation graph contains computation graphs of them as well.
Of course information of feature_adversarial would be retained if you use loss.backward(retain_graph=True), which is wasteful.
But in my personal opinion, maybe pytorch will destroy computation graph of feature_adversarial of i-th iteration when new feature_adversarial is constructed on i+1 th iteration.