If someone face similar issue, the problem may be caused by cudastreamsync() when transfer minibatch generated by dataloader from CPU to GPU. Since the tensor transfer is on the default cuda stream, this forces an addition synchronization in every iteration. The issue can be solved by put the tensor transfer on a separate cuda stream.