Regarding the performance, could you set the number of threads to 1 and see if it is still slower?
Regarding the noise - Could you try with pytorch nightly build? There was a fix for the scale issue as mentioned here - https://github.com/pytorch/pytorch/issues/33466#issuecomment-627660191 6