Based on my understanding, PyTorch provides two APIs for profiling our application.


One is the torch.autograd.profiler.profile API. It has use_cuda flag, and we can choose to set it for either CPU or CUDA mode.


Another API for profiling is torch.utils.bottleneck (which provides both CPU and CUDA mode profiling according to the documentation https://pytorch.org/docs/stable/bottleneck.html 15).


And I expect them to give same or at least very close profiling results, but they actually didn’t. Below are my snippet for profiling and results I got.
def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('-o', '--option', default=1, type=int)
    args = parser.parse_args()

    data = torch.randn(1, 3, 224, 224).cuda()
    net = resnet18().cuda()
    net.eval()

    # Warm up run
    for _ in range(10):
        net(data)

    if args.option == 0:
        for _ in range(100):
            out = net(data)

    elif args.option == 1:
        # Profiling 1
        start = torch.cuda.Event(enable_timing=True)
        end = torch.cuda.Event(enable_timing=True)

        timing = 0
        for _ in range(100):
            start.record()
            out = net(data)
            end.record()
            torch.cuda.synchronize()
            timing += start.elapsed_time(end)
        print(timing)

    elif args.option == 2:
        # Profiling 2
        with torch.autograd.profiler.profile(use_cuda=True) as prof:
            for _ in range(100):
                net(data)
        print(prof.key_averages().table(sort_by='cuda_time_total'))
    elif args.option == 3:
        # Profiling 3
        with torch.autograd.profiler.profile() as prof:
            for _ in range(100):
                net(data)
        print(prof.key_averages().table(sort_by='cpu_time_total'))

torch.util.bottleneck API
python -m torch.utils.bottleneck prof.py -o 0

CPU mode ... 
Self CPU time total: 218.316ms
CUDA time total: 0.000us

CUDA mode ...
Self CPU time total: 207.595ms
CUDA time total: 207.673ms

torch.autograd.profiler.profile API
python prof.py -o 3

CPU mode ...
Self CPU time total: 295.712ms

python prof.py -o 2
CUDA mode ...
Self CPU time total: 499.643ms !!!!!!
CUDA time total: 1.802s !!!!!!

My question is why there is such a big profiling time discrepancy when I use torch.autograd.profiler.profile(use_cuda=True)?