Thanks for the replies. I will give more detail. First of all, I am talking about CPU RAM not GPU. My pipeline is a simple autoencoder and I have 500k images. Batches consist of 512 images. I fetch data from a local disk with a custom dataset class and then I use the dataloader from torch.utils:

    loader = torch.utils.data.DataLoader(dset,
                  batch_size=args.batch_size, shuffle=True,
                  num_workers=args.workers, pin_memory=True)


The fact is that I can train this model but I need to ask for a huge amount of CPU RAM even though I would expect the memory load to  be constant throughout one epoch of training. Instead I observe a linear increase in memory, a memory leak I guess. Here is a graphic of the memory load:

image.png1247×249 14.4 KB

Each one of those cycles is an entire epoch. As you can see, the memory clearly increases throughout and I need more than 200GB of RAM otherwise I run out of memory.
I then stripped my code of all the unnecessary and now my train function that is called per each epoch looks like this:

def train(loader, epoch):
    for i, (input) in enumerate(loader):
        print('Epoch: [{0}][{1}/{2}]\t'.format(epoch+1, i+1, len(loader)))


Even like this the memory usage increases linearly as batches are loaded.
I have tried manually deleting variables and calling garbage collection to no avail.