Hi,
I am wondering what the expected time for kernel calls is. PyTorch kernel calls are asynchronous, so the GPU will do work while the CPU can already launch new kernels. I would expect something like the forward pass to very quick in python as long as there are no synchronization points. Obviously, when I need the result, I would have to wait for the CPU.
However, when I benchmarked this by simply measuring time in python for different operations it seemed like these were blocking. E.g.
-prediction.log_prob(target).mean()

takes longer than the forward pass of my model for some reason.
Any idea whether this is to expect?