del checkpoint  # dereference seems crucial

Worked for me !
It seems to have saved around +500mb in my case!
I was able to save but not load in pytorch. The checkpoint definitely took up valuable valuable gpu memory.
torch.cuda.empty_cache() 

Helped clear around 300mb!
Thank you @roytseng  !