Gotcha, so what sort of circumstance would lead to the GPU blocking the copy? I know pinned memory is recommended for data loading directly to the GPU, but it’s still not abundantly clear to me how it helps.
Is it primarily a way for the data loader to prefetch the next batch onto the GPU while the current batch is being processed? For example, if you have a network architecture that performs some inference task on the GPU. Without pinned memory, execution would be:

Load batch to GPU
Execute inference
Load next batch to GPU
…

Do I understand correctly that with pinned memory, we would have

Load first batch to GPU
(concurrent with 3) Execute inference
(concurrent with 2) Load next batch onto GPU
…

Is that the general idea? If so, does it only execute the asynchronous copy if there is enough GPU RAM to accommodate it?