Solved by SimonW in post #5 
              
              
                Yeah, here is what happens: 

If len(LG_semi_arr.data.cpu().numpy()) != 0, then LG_semi_arr is included in the computation graph of LG_seg. So when you call LG_seg.backward(), the relevant graph part is freed.
If len(LG_semi_arr.data.cpu().numpy()) == 0, then the graph backing LG_semi_arr can’t be f…