Hello, thank you for your reply.
I have wrongly used “autograd” to refer to the general backwards propogation pass (error and gradient). The autograd is something rather mysterious to me; something that will require much more attention.

Are you interested in the first or second in your case?

I am interested in the second case.
When refering to GPU memory usage, we generally have the problem where memory is full of “intermediate” tensors and a OOM event occurs. This is closely linked to which tensors are generated by forward propogation, stored in memory and then reused for error and gradient calculation.
That construct looks somewhat promising - I will take a look over the next week.
I was also thinking about potentially using a tensor’s version counter 1. Each access could trigger a function call here that may be able to record some metadata for my use.