Solved it.
For those that come up with this problem here are the two steps I used to solve it.

Make sure that your conda is updated to the latest version

In my case this was already done, but just make sure it is in your case.

temporarily move your ~/.condarc file if you have one.

For me I just removed it, it turns out that I had a bunch of repos listed there, that I believe messed with the priority search path for packages. Not sure what the priority is when you provide the “-c” flag to an install command and you have repos listed in your condarc file. I suspect the file takes precedence.
After the above, I now have a pytorch 0.3 cuda installation