I have a Variable(FloatTensor) in my graph that is adding another Variable(FloatTensor) to it but on every iteration of forward in the custom nn.Module it allocates memory that is never deleted.  Is there some other way I should be writing this line?  Or is there a way to manually delete the allocated memory for the addition?  The following line seems to cause the problem:
var1[:,index1,:,:] = var1[:,index1,:,:] + var2[:,index2,:,:]

using objgaraph it I get the following names for the leaked items:
before - top 2:
tuple                      3191
dict                       3037
after 1000 runs, top 4:
slice                      72076
Size                       50036
dict                       47093
tuple                      28216
and categories increment every time they pass over that line.  Which tracemalloc also pointed me towards.