Solved by ptrblck in post #2 
              
              
                The first CUDA operation will create the CUDA context, which contains the the PyTorch kernels, cudnn, NCCL, CUDA libs etc., so it’ll take some time to load these libraries. 
Also note, that CUDA operations are executed asynchronously, so if you want to profile the data transfer from host to device o…