Hey thanks for the thorough explanation.
Could you please clarify what do you mean by “overlap compute and memory transfer to the GPU.”? I understand what’s referred to here by “memory transfer” (asynchronously) , but what does the “compute” part refer to here?
Does that mean that the model (kernel) will start its execution process on the input data while the target is being transferred to GPU memory?