Unsure, if it’s missing from your code, but if you are working on an inference use case, you might want to warp the forward pass into a with torch.no_grad() block to avoid storing intermediate tensors which would be needed for the gradient computation.
Could you add it and check, if the memory usage goes down?