Your code should generally work to free all allocated memory so I guess you might be missing to delete all references to all tensors stored on the GPU.
Once this is done, note that the CUDA context will still use memory to hold all the kernels which might take up to ~1GB depending on the device, CUDA, PyTorch version etc.