I was able to free probably all the GPU memory used by tensors by using the following sequence:

model.to(‘cpu’)  # this allows moving to area where you probably have more memory
model_RAM_copy = model.state_dict()
delete_tensors()  # this function goes through model children and deletes .weight, like del model.layer1.weight; I do not care about .bias, they are much smaller
torch.cuda.empty_cache()
model.to(‘cuda:0’)
model = newModel().to(‘cuda:0’)  # recreating the model structure from scratch
criterion =… optimizer =…, lr_scheduler =…
torch.cuda.empty_cache()
model = load_state_dict(model_RAM_copy )

In fact, my code is a little longer, but I think there is some redunduncy, I just didn’t take time to optmize it. Anyway, it works in terms of memory usage (I can see a beutyful line of GPU Memory usage going up and down, within some stable limits). I am sure this can be done more smoothly, but I was looking for a solution for few days and ended up with this. Does not look pretty, but works. I am not sure about backpropagation correctness or preserving learning rate statistics. Network learns pretty ok, but it is possible that I am loosing some parameters on the way.