Right, and I don’t have the memory leak when I run the code on the CPU, only when I switch to the GPU.
I don’t think this is related to that particular bug, because my leak happens in the forward pass (that issue is specifically about .backward()). My memory leak occurs anytime I accumulate tensors. “Accumulate” could mean, e.g.: assigning a tensor at the index of another tensor (lines 49-52), iteratively summing losses with +=, or appending a tensor to a list. Somehow this accumulation prevents the tensors from being garbage collected.
These are the details of my hardware, drivers, and software:

OS

Ubuntu 16.04.5 LTS


GPU

NVIDIA Quadro GP100


CUDA

9.0.176


cuDNN

6.0.21


PyTorch

0.4.0



If this problem comes from below the code as you’re suggesting, is there a short-term fix I can use in the code to get around the memory leak without disrupting backprop?
Perhaps copy.copy() is copying too shallowly, leaving out some important attributes that make backprop work. Do you know what attributes those might be?
Given this hypothesis of shallow copying being the culprit for the backprop bug, I’ve tried to use copy.deepcopy() for lines 49-52. But using copy.deepcopy() leads to the following error:
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment