Hi, first of all thank you for reading this question. I have a script (say main.py) where I train multiple models (with pytorch lightning) with different hyperparameters. The loop in the script is similar to the following code snippet:
for hparam in HPARAMS:
     trainer = Trainer(gpus=gpus)
     datamodule = Datamodule()
     model = Model(hparam)
     trainer.fit(model, datamodule)

After training the second model, I get a memory leak with the gpu. I have seen this question a couple of times in forums (sorry in advance) and no suggested solution worked for me.
I tried deleting all relevant variables that could store tensors with
del model, datamodule, trainer, logger
gc.collect()
torch.cuda.empty_cache()

but this did not fix the memory leak. I did this after every training of a model.
I read a couple of other suggestions like using ray or just starting a new subprocess for every new training but I thought there must be another way.
Any help is much appreciated!