For NAS (Network Architecture Search) I need to measure the latency of operation that are present in my search space.
Therefore, I tried different approaches to measure the latency of a nn.Module:

pytorch autograd profiler
“normal” time.time() measurements
cuda events

Of course I used torch.cuda.synchronize to account for any asynchronous execution.
Also I averaged all my measurements over >10000 of data points.
My problem now is, that I can’t get get:


Reliable measurements: nn.Conv2D(48,48, k=3) called with torch.randn(1,48,48,48) is sometimes slower than when I call the module with torch.randn(1,48,24,24). The same goes for using different channels: Higher channels should result in higher latency but sometimes the opposite is the case.

Repeatability Almost always, a second execution of the test script gives different results. Remember that I used torch.cuda.synchronize everywhere its needed.

A reason for 1. could be the massive parallelism of modern GPUs: The GPU just doesn’t care if the convolution uses 128 or 256 channels. It will calculate all results in parallel anyway, resulting in more or less the same latency.
For 2. I don’t have any clue. Why is the behavior of my GPU differently at different times? This makes all NAS papers or other papers that claim to have SOTA latency basically absolutely unreliable.
tl;dr:
Is there any method to measure module latency in a reliable and reproducible way?