There’s a lot going on here. First, you’re not going to see an 85x (or 10x) speed-up here at these sizes. GPUs have a lot more compute (FLOPs) than memory bandwidth. These bottleneck convolutions trade-off flops for memory bandwidth (each convolution has to read the entire input and write the entire output). In the baseline cases, the compute dominates memory access, but that’s less true in the bottleneck layers. (This part isn’t a matter of CPU overhead).
Second, GPU applications, including PyTorch rely on hiding CPU overhead by executing GPU operations asynchronously. The CPU and GPU operations overlap, so as long as the CPU overhead take less time than GPU operations (and synchronizations are infrequent enough) the program is limited by the GPU. It looks like you’re synchronizing after every layer, which results in the overhead being added instead of overlapped. The fix is move the synchronization outside the loop in benchmarking code.
Third, profilers (nvprof and autograd profiler) add CPU overhead. They can be really useful for determining the execution time of GPU kernels, but may be too pessimistic in terms of estimating CPU overhead.
Fourth, it looks like you are running with batch size 1. If you want a performance improvement, batch your inputs. This can help in two ways: the GPU kernels are often more efficient with larger batch size and it amortizes overhead across multiple inputs.
I ran your code with my script.py 3. I ran it both with and without nvprof on a P100. The results I got are below. “Total Time” is the time printed by the script. “GPU kernel time” is the sum of the average execution times of the kernels reported by nvprof.
It looks like the CPU overhead matters in “A - Distilled” case, but not the others. It might not be a problem if the rest of your network is not CPU bound (due to latency hiding and asynchronous execution). What else can you do about it? Well, that depends on your use case (inference-only? can you batch?)




Example
Total Time
GPU kernel time




A - Baseline
124 us
122 us


A - Distilled
123 us
71 us (***)


B - Baseline
214 us
207 us


B - Distilled
99 us
96 us