Could you explain your profiling and use case a bit?
Also, are you using the same input shapes?
Is the model using the CPU or GPU? In the former case, you might see a performance hit due to the processing of denormal values, which is slower on the CPU. In that case, try to set torch.set_flush_denormal(True) and rerun the code. If you are using the GPU, make sure to synchronize the code properly via torch.cuda.synchronize() before starting and stopping the timer.