Hi,
I am working on benchmarking models for a future update in production, and I noticed a considerable speed reduction between 1.2 (the version we currently use) and 1.7 (the version we wish to use).
I ran tests with 1.2, 1.3, 1.4, 1.5, 1.6 and 1.7, always with CUDA 10.2 and cuDNN 7.6. PyTorch was install ed with pip3 on a Ubuntu 18.04 running the AWS Deep Learning AMI on a p3.2x which uses a V100-SMX2. Installed CUDA version is 11, driver version is 450.80.02.
Now for the actual numbers, 1.2 to 1.5 clock in at about 80ms/batch, 1.6 does 147ms/batch and 1.7 does 120ms/batch. All measurements were made with torch.cuda.synchronize() after each iteration for 100 iterations. Timings above were measured with time.time().
Unfortunately I cannot distribute the actual code for the model, but it’s a pretty standard feature extractor with an attention layer on top.
Finally, when benchmarking with the PyTorch profiler, the numbers are very different (and make more sense):

PyTorch 1.2: CPU time: 97ms CUDA time: 315ms
PyTorch 1.7: CPU time: 130ms CUDA time: 131ms

Has anyone encountered a similar issue, a <10% increase would be fine but this is +50%?
Thank you
Edouard