Did you synchronize the code properly before starting and stopping the timer?
If not, this would yield invalid profiling results, since CUDA operations are asynchronously executed and your Python script might “run ahead”.
Also note that you shouldn’t expect speedups on the P40, as no TensorCores are available on this device.