How do you profile your execution time? The most operations in Pytorch are async, which returns right after being called. So it makes sense to add torch.cuda.synchronize() before and after your critical code lines when you are profiling.