Can you make sure that the cuda version of the pytorch and cuda version in PATH variable (or) /usr/local/cuda are same?
I faced similar issue. I am not sure if this was the root cause. But make sure of this.