Solved by MariosOreo in post #7 
              
              
                It seems there are some variables that calculated in global and used for backpropagation. But in my opinion, the large amount memory cost is caused by retain_graph that calling backward() cannot clear the graph in each iteration. As a result, memory used is gradually increasing. 
Maybe the solution …