One more question: what is Pytorch’s unit for enqueuing computations to the CUDA stream?

Not sure what you mean by unit. But usually each op corresponds to a single kernel. With few complex ones launching few kernels.
It will depend on the op I’m afraid.

But I couldn’t imagine a way where a TensorImpl::function could be run when a tensor is accessed.

There is no check for “accessing” a Tensor.
That is why I recommended to attach to the autograd construct that save/unpack the Tensors for the backward pass.