Again, really appreciate your help!  Your answer inspired me.
torch.no_grad() actually works, the intermediate tensor is not in GPU anymore. However, the occupancy does increase a lot. I find that part of memory is cached.
And using
torch.cuda.empty_cache()

can release the memory.