There is something wrong with your drivers or with you compilation if you compile Pytorch yourself. I am not sure what the real reason is, I just remember, that I had the same problem, when I messed up my installation!