Thank you so much for the suggestions! After adding your two lines of code,  the allocated memory is still:
After stream s before graph g, allocated:  13397522944 ; reserved:  14986248192.

I think I need to figure out why the memory is not clearing. Is this the reason why the reserved memory doubles during capture graph?
The model is a quantum simulation model, which generates enormous tensors and uses torch.einsum everywhere with complicated python logic. This is why it gets so large. The simulator itself is a black box and I gotta dig deeper.