There is also an approach which I though should work to resolve it but having issues while implementing it too(Pasting from Stack Overlfow)-
I am trying to install Pytorch in a remote server with no root privileges.
I want to install Cuda 10 on it. I have installed conda environment on it already.
Remote server has already few cuda versions like cuda9.1 and older than it.
After installing cudatoolkit:10.1.168 in the conda environment, I don’t know how to use this cuda in place of already present in the remote machine.
It use by default the already installed cuda versions by the system(9.1 in this case).
I had the same issue regarding using the python of conda which I resolved using 
unset PYTHONPATH

Is there an equivalent or any other way to use conda’s cuda?

0S-CentOS 6.5
Conda-4.3.27
Conda's python - 3.6.5