I see that you might be using backward hooks from here 4 if debug is enabled.
Are you manipulating these tensors somehow? If not, are you seeing the issue without using the hooks? Would it work to detach the tensors before appending them to the global list (assuming that fits into your use case)?