I’ve had a similar problem while doing active learning.
Here is what I found. Deleting the model, datamodule, trainer and logger does nothing to solve the gpu memory leak. After much investigative work, I found out that the optimizer (I was using torch.optim.Adam) stores pointers to the model’s weights and it has some other tensors of its own that also resides on the gpu. I wrongly believed that deleting both the model and the trainer would delete the optimizer object.
In my case, my problem was solved in two step:
# Assumes that your pytorch-lightning Model object
# has the pytorch model object as self.model
model.model.cpu()

to remove all the model’s weights from the gpu.
Then, we need to update the optimizer’s internal tensors and bring them out of the gpu. I did it the following way but I would like a more official way supported (does it already exists?).
# I only have Adam in my project but this may be different in your case.

for optimizer_metrics in trainer.optimizers[0].state.values():
    for metric_name, metric in optimizer_metrics.items():
        if torch.is_tensor(metric):
            optimizer_metrics[metric_name] = metric.cpu()


After these two steps, I am still left with one tensor residing on the gpu but I can’t find where it is still alive. Everytime I go through the loop, one extra tensor still resides on the gpu. That extra tensor per loop isn’t enough to make me bust my cuda memory but I would like to hear back if someone figures out where it is referenced.