Hello,
I am struggling with implementing reversible residual network.
When I implement this, the computational graph is consisted of some nodes, and this nodes are constructed using Tensor.
The strength of reversible network is that it can construct earlier activation with current activation. However, the computational graph use Tensor as nodes, which makes them always store the earlier activation.
I want to use gpu memory as efficient as I could. So, I want to retain the structure of graph while the tensor data is freed. Can I do this? If I can, how to do this?
One possible solution is to delete tensor.data. However, I worried that it makes unexpected result.
Thanks in advance.