I just don’t like the idea of doing this. I’m more of favor of telling tensors to release ownership to the pointers.

The problem is not with the ownership but with how will you know how to free the memory?
We use different allocators depending on what you do (regular cpu memory, regular gpu memory, shared cpu memory, paged cpu memory,…). And you won’t be able to call just free or delete on the pointer to clean it properly!
Do you think it would be possible for your ManagedMemory to have an optional Tensor field that will make sure it keep the dataPtr alive? And when the ManagedMemory is freed, the Tensor will go away and free the memory.