You are still bound to have written some glue code that deals with tensors?
Generally, those graphs get unwieldy much faster than just the inputs.
One of the things that easily happens is to not keep the leaf tensors:
x = torch.randn(1, 2, 3, requires_grad=True).cuda() is not a leaf tensor, so computing something with x, doing backward, computing something with x and then doing backward again, it will try to backward twice through the .cuda() step.
Best regards
Thomas