Hi,
I am training a model and I saw that the GPU usage increases epoch after epoch, even by calling torch.cuda.empty_cache().
Here is a snippet of code:
for sample_id, (packed_blocks, packed_sequences) in tqdm(enumerate(data_loader)):
    monitored_quantities = step(packed_blocks, packed_sequences, train=train)

    means = {
        key: value + means[key]
        for key, value in monitored_quantities.items()
    }

    torch.cuda.empty_cache()

Before the first loop, GPU memory usage is 489MB. I assume it corresponds to the model stored in GPU. But at the beginning of each new loop, the GPU memory is higher and higher. And yet:

forward and backward pass are computed inside of step function

packed_blocks and packed_sequences are on CPU when the dataloader drops them, they are loaded on GPU inside of step function

monitored_quantities and means are dicts of floats

Since no new CUDA tensor is created in the scope of the for loop, I assume that after calling torch.cuda.empty_cache() the GPU memory usage would be the same than at the beginning of the iteration. But in practice it keeps growing.
Does anyone have an idea about this question?
Best,
Alain