Is it because of the multi-threaded data loading to GPU? https://pytorch.org/docs/stable/data.html#working-with-collate-fn 97
It seems the process of moving data to GPU should not be done asynchronous. Hope this can help .