I’m not sure why this should be caused by a memory leak.
The backward pass will use some memory to store all gradients.
Depending on your model architecture and thus the shape of these gradients, the memory might increase by a large amount.
If you are seeing the OOM in the first iteration(s), then this is most likely the cause and you could try to use e.g. torch.utils.checkpoint to trade compute for memory.
However, if you are seeing the OOM after e.g. an epoch, then you might want to check if you are accidentally storing the computation graph somewhere, e.g. via losses.append(loss) (you would have to call loss.detach() in order to avoid storing the whole graph).