I found the cause. Because I want to randomly init leaf_h and set it as constant. So I set leaf_h as Variable and save into class attribute. Thus, for every sample, leaf_h (which is Variable) will log it creator. Therefore, for every computation graph which leaf_h participate in, backward get longer. Leaf_h is class attribute, so it is not destroy for every sample or batch, but stay there until end. Thus, the “history” in leaf_h is not destroy.
My suggestion for fixing this:


Save self.leaf_h as torch.Tensor
self.leaf_h = torch.rand(1, self.mem_dim)


When ever need to use leaf_h, init another temporate Variable
temp_leaf_h = Var(leaf_h)