the GPU memory of all tensors that have been referenced directly or indirectly should be available to get reused?

Yes exactly.

My problem is that I expected this to happen but it does not appear to happen

Could you do a small code sample (30-40 lines) that reproduces this please?