Yeah, here is what happens:

If len(LG_semi_arr.data.cpu().numpy()) != 0, then LG_semi_arr is included in the computation graph of LG_seg. So when you call LG_seg.backward(), the relevant graph part is freed.
If len(LG_semi_arr.data.cpu().numpy()) == 0, then the graph backing LG_semi_arr can’t be freed as you still have reference to LG_semi_arr until you reach this part in the next iteration!

Here is what I would do:
if LG_semi_arr.size(0) != 0:
  LG_semi = -1*LG_semi_arr.mean()
else:
  del LG_semi_arr
  LG_semi = Variable(torch.zeros(1).cuda())  # just write 0 here should also work

Note that I also rewrote your condition so you don’t need to pull it to CPU and convert to numpy ndarray.