The memory increases progressively with each 500-1000 iterations batches until it reaches the max GPU capacity. gc.collect, cuda.empty_cache(), deleting the loss, input, target after the optimizer.step and loss is added to the losses[] array, none of them work.