I had a similar problem and Clement indicated this tutorial helps install it.
0.1 – Install PyTorch on OSX
The problem as already mentioned seems to be the xcode version not compatible with the version of cuda and a downward is necessary. Good luck