I solve this problem now!!~~~
I installed pytorch from source from the beginning, and install magma-cuda80.
And later I installed torchvision with conda
conda install torchvision -c soumith
it forced me to update to pytorch-cuda75 …
Now I install both pytorch and torchvision from source, problem solve