Hi,
You can install pytorch without conda from source (I never did it with conda actually).
For best performances, make sure you have the latest cuda and cudnn installed and OpenBLAS if you’re going to do some CPU stuff.
Thinking about it, the difference could come from a different cudnn version bundled in pip and in your conda install.