I’ve been trying to pin down the issue behind the constant increase of GPU usage but I’ve been unable to.

image.png1030×478 48.3 KB

[it keeps increasing until the kernel dies]
I’ve tried solutions that were given before (del output, output_max, loss, data, targets, index, inputs, labels, make sure I use loss.item(), gc.collect(), torch.cuda.empty_cache()) but nothing worked. What is the best way to track down what’s causing this?
Not sure if this is relevant but one of the things that is different in my implementation is that although each batch is accessed initially using the dataloader, the dataloader also returns the indixes of the accessed data which are then used in conjunction with getitem(index) function to access ‘subsets’ of those data a couple of times.
Let me know if you need further information.