Ideally, it would be for all tensors, implicitly and explicitly created by the user. That means p, q, r and all the other tensors produced by subfuctions.
Originally, I thought I would be able to augment the TensorImpl structure, so that each unique tensor could track its own access pattern (access count and timestep). But I couldn’t imagine a way where a TensorImpl::function could be run when a tensor is accessed. I also wondered if tensor accessors could possibly provide similar functionality…
One more question: what is Pytorch’s unit for enqueuing computations to the CUDA stream?