How much memory is being occupied? If it is around 500mb it is the CUDA context cache.
You can simply move a model to the CPU using model.to(torch::kCPU) to release the GPU memory.  Just make sure to call c10::cuda::CUDACachingAllocator::empty_cache() afterwards. You may need to move the model to the CPU within a function and call empty_cache() afterwards.