Based on the versions, it looks like you are using the PyTorch binaries.
If that’s the case, your local CUDA installation won’t be used, but the one shipped with the binaries.
Could you install the same binaries with matching versions and, if possible, post your code for profiling so that we can have a look, please?